Logtash-Forwarder 迁移到 Filebeat
admin- 首页
- 证书相关
- Logtash-Forwarder 迁移到 Filebeat
Logtash-Forwarder 迁移到 Filebeat
发布日期：2017-04-19
Filebeat带来下面的变化：
- 对配置文件格式进行了重组，从JSON转换为YAML。
- 对存储当前读取文件的状态的registry file被改变。
- 命令行选项被删除并移到配置文件中。
- 输出的配置选项从libbeat继承。
- Logstash必须使用一种新的输入插件。
迁移策略
logstash
Logstash 需要安装一个新的输入插件 logstash-input-beats。在Logstash 1.5.x版本和2.x版本，该插件可以与 Logstash-Forwarder 所使用的插件logstash-input-lumberjack 并行加载。
如果你有大量的logstash-forwarder迁移到Filebeat，建议同时加载这两个插件，将其设置为不同的端口。当所有的迁移到Filebeat，即可删除Lumberjack插件。
Registry File
Registry File存储了Filbeat最后一次读的位置和状态。在Logstash-Forwarder被称为.logstash-fowarder(位于/var/lib/logstash-forwarder/.logstash-forwarder)。对于Filebeat需要将其重命名为 .filebeat。
迁移配置文件
files部分
Logstash-Forwarder 配置文件中的”files”部分转变成Filebeat配置文件中的”prospectors”部分。如：
logstash-forwarder配置文件
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | # The list of files configurations “files” : [ # An array of hashes. Each hash tells what paths to watch and # what fields to annotate on events from those paths. { “paths” : [ “/var/log/messages” , “/var/log/*.log” ] , # A dictionary of fields to annotate on each event. “fields” : { “type” : “syslog” } } , { # A path of “-” means stdin. “paths” : [ “-“ ] , “fields” : { “type” : “stdin” } } , { “paths” : [ “/var/log/apache/httpd-*.log” ] , “fields” : { “type” : “apache” } } ] |
相当于Filebeat配置文件中的prospectors部分：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | filebeat : # List of prospectors to fetch data. prospectors : # Each – is a prospector. Below are the prospector specific configurations – paths : – / var / log / messages – “/var/log/*.log” – paths : – “-“ input_type : stdin document_type : stdin – paths : – “/var/log/apache/httpd-*.log” document_type : apache |
引入了一个新的选项document_type，如果没有类型被定义则默认为log。如果Filebeat被直接用来Elasticsearch索引，那么当在索引时document_type决定文档类型。
network部分
Filebeat可以于Logstash直接通信，此外，Filebeat还可以直接向elasticsearch插入日志条目。
logstash-forwarder配置文件：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | # The network section covers network configuration “network” : { # A list of downstream servers listening for our messages. # logstash-forwarder will pick one at random and only switch if # the selected one appears to be dead or unresponsive “servers” : [ “trustauth.cn:5043” ] , # The path to your client ssl certificate (optional) “ssl certificate” : “./logstash-forwarder.crt” , # The path to your client ssl key (optional) “ssl key” : “./logstash-forwarder.key” , # The path to your trusted ssl CA file. This is used # to authenticate your downstream server. “ssl ca” : “./logstash-forwarder.crt” , # Network timeout in seconds. This is most important for # logstash-forwarder determining whether to stop waiting for an # acknowledgement from the downstream server. If an timeout is reached, # logstash-forwarder will assume the connection or server is bad and # will connect to a server chosen at random from the servers list. “timeout” : 15 } |
Filebeat相当于：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | output : logstash : enabled : true # The list of downstream Logstash servers. hosts : – trustauth.cn : 5043 tls : # The path to your SSL client certificate. certificate : . / logstash – forwarder . crt # The path to your SSL client certificate key. certificate_key : . / logstash – forwarder . key # The path to your trusted SSL CA file. This is used # to authenticate your downstream server. certificate_authorities : – . / logstash – forwarder . crt # Network timeout in seconds. timeout : 15 |
当定义多台主机，类似于Logstash-forwarder行为，Filebeat默认随机选择一个主机建立连接。Filebeat可用设置为负载均衡。参见：https://www.trustauth.cn/guide/en/beats/libbeat/1.0.0-rc1/configuration.html#loadbalance
更改后的配置文件选项
配置文件的重构，有些选项被删除或改名。下面是更改的条目列表：
| Config Option | Action |
|---|---|
deadTime |
deadTime was renamed to ignoreOlder. In case a file is not changed for ignoreOlder, the file handler will be closed. If the file is changed again after ignoreOlder has passed, it is be reopened. |
netTimeout |
netTimeout was removed as it is replaced by the Timeout option in libbeat. |
log-to-syslog andsyslog |
Both options were removed as logging is part of the libbeat config. |
完整的实例
Logstash-Forwarder配置文件
| 1 2 3 4 5 6 7 8 9 10 11 12 13 | { “files” : [ { “paths” : [ “/var/log/*.log” ] , “fields” : { “type” : “syslog” } } ] , “network” : { “servers” : [ “trustauth.cn:5043” ] , } } |
Filebeat配置文件：
| 1 2 3 4 5 6 7 8 9 10 11 | filebeat : prospectors : – paths : – “/var/log/*.log” fields : type : syslog output : elasticsearch : enabled : true hosts : [ “http://trustauth.cn:5043” ] |
命令行选项
大部分 logstash-forwarder命令行被删除并移到配置文件中，重命名的命令行选项列表如下：
| Command Line Option | Config File Option | Description |
|---|---|---|
-config |
-c |
The config options was split up in two part. The base and required config is linked with -c. Additional config files can be linked as part of the config file. Note: Additional config files must be in a different directory than the main config file. |
-config |
config_dir |
Path to directory with additional configuration files |
-idle-timeout |
idle_timeout |
idle_timeout was moved to the config file and removed as flag. |
-spool-size |
spool_size |
spool_size was moved to the config file and removed as flag. |
-harvester-buff-size |
harvester_buffer_size |
harvester_buffer_size was moved to the config file and removed as flag. It can now be configured specific for each harvester. |
-tail |
tail_files |
tail_files was moved to the config file and removed as flag. It can now be configured specific for each prospector. |
-cpuProfileFile |
cpuProfileFile option was removed. The profiling options of libbeat can be used instead. For more details on profiling see https://github.com/elastic/libbeat/issues/122 |
|
-quiet |
The quiet option was removed. Libbeat is used for logging and the libbeat configuration options have to be used. |
其它的一些改变
- 包
一个显着的变化是registry file名称取决于包封装类型：
.tar.gz 和 .tgz 名称为.filebeat
DEB 和 RPM 名称为/usr/lib/filebeat/registry
Windows zip包 名称为c:\ProgramData\filebeat\registry - TLS默认是关闭的
- 日志
Filebeat使用libbeat日志，也可以记录到轮滚的文件，而不是系统日志。
文章转载来自：trustauth.cn
上一篇：解决could not create the java virtual machine问题
下一篇：使用 Vagrant 打造跨平台开发环境
相关新闻
- SSL证书对网站访问速度有影响吗
- 个人隐私数据泄露有哪些危害？如何预防？
- 部署SSL证书有哪些常见的错误
- 国际证书、国产证书和国密证书
- 游戏开发为什么离不开代码签名？
- 僵尸网络攻击手法与防范方式
- SSL证书助力保障网络数据安全
- 网站加密与不加密区别
- SSL证书有哪些类型和价格差异
- ca机构颁发的证书包括那些内容呢？
