![[PukiWiki]](image/owl.png "[PukiWiki]")
現象:
# jsonの構文チェック aws s3 cp s3://bucket/path/to/example.gz - --profile example | zcat | jq . 1>/dev/null;echo $? # 最終行のみ表示, 指定行の場合 sed -n '100p' aws s3 cp s3://bucket/path/to/example.gz - --profile example | zcat | tail -n 1 # 最終行の16進数表示 aws s3 cp s3://bucket/path/to/example.gz - --profile example | zcat | tail -n 1 | od -tx1c
原因？
scrape_configs: - job_name: 'fluent-bit' metrics_path: /api/v1/metrics/prometheus ec2_sd_configs: - port: 2020 filters: - name: tag:site values: ["dev"] - name: tag:role values: ["web"] relabel_configs: - source_labels: [__meta_ec2_instance_state] regex: 'stopped' action: drop - source_labels: [__meta_ec2_tag_Name] target_label: instance_name - regex: '__meta_ec2_tag_(site|role)' replacement: '${1}' action: labelmap
garafana
grafana var:
dashbord:
# fluent-bitのendpoint総数
count(up{job="fluent-bit",site="$site"})
# 有効なfluent-bitのendpoint数
count(fluentbit_uptime{site="$site"})
# Input Bytes Processing Rate
code: sum(rate(fluentbit_input_bytes_total{site="$site", instance_name=~"$instance_name"}[5m])) by (instance_name, name)
legend: {{ instance_name }}/{{name}}
複数の方法がある。
記事:
[OUTPUT] Name kinesis_firehose Match * region ap-northeast-1 delivery_stream example1 compression gzip # Non UTF-8 record provided. Only UTF-8 encoded data supported
記事:
[INPUT]
Name tail
Path /var/log/messages
Path_Key logpath
{"logpath"=>"/var/log/messages", ... }sudo cat /var/log/messages | perl -ne 'BEGIN{$max=0;$line} $len=length; if($len > $max){$max=$len;$line=$_;} END{print "$max," . substr($line,0,512) . "...\n";}'
[PARSER] Name syslog Format regex Regex ^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S Time_Keep On
date --iso-8601=sec 2022-09-07T09:50:32+0000 # アプリによっては%SZ, %S.%LZが混合している場合もあった。 2022-09-07T09:50:32Z 2022-09-07T09:50:32.000Z
[PARSER]
Name access_log_ltsv_iso_8601
Format ltsv
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S
[FILTER] Name record_modifier Match input.* Record host_name ${HOSTNAME}
[FILTER]
Name aws
Match input.*
imds_version v1
az false
ec2_instance_id true環境:
[SERVICE] flush 5 grace 30 parsers_file /etc/fluent-bit/parsers.conf [INPUT] Name tail Path /var/log/messages Tag input.syslog.messages Parser syslog-rfc3164 DB /opt/fluent-bit/state/input.syslog.messages.db Mem_Buf_Limit 5MB Rotate_Wait 30 Skip_Long_Lines On [OUTPUT] Name stdout Match *
sudo /opt/fluent-bit/bin/fluent-bit -c ./input_syslog_messages.conf
echo test | logger -i -t "tag.test"
echo "$(date "+%b %d %H:%M:%S") $(hostname -s) test[$PPID]: test message" >> /tmp/test1.log
記事:
記事:
FLB_VERSION=2.0.8 # replace old yum repo if [ ! -f /etc/yum.repos.d/CentOS-Base.repo.bak ]; then cp /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak; fi sed -i 's/^mirrorlist=/#mirrorlist=/g' /etc/yum.repos.d/CentOS-Base.repo sed -i 's|^#baseurl=http\:\/\/mirror.centos.org|baseurl=https://vault.centos.org|g' /etc/yum.repos.d/CentOS-Base.repo # build requirements yum -y install epel-release yum -y install git cmake3 zlib-devel yum -y groupinstall "Development Tools" # bison cd if [ ! -d bison-3.4.2 ]; then curl http://ftp.gnu.org/gnu/bison/bison-3.4.2.tar.gz | tar zxf - cd bison-3.4.2 ./configure --prefix=/opt/bison-3.4.2 && make -j 4 && make install fi # fluent-bit cd git clone https://github.com/fluent/fluent-bit cd fluent-bit/build git checkout v${FLB_VERSION} PATH=/opt/bison-3.4.2/bin:$PATH cmake3 -DCMAKE_INSTALL_PREFIX=/opt/fluent-bit/fluent-bit-${FLB_VERSION} .. make -j 4 make install
docker run --rm -it -v $(pwd):/mnt centos:6 bash cd bash /mnt/build.sh
