01-docker log管理和时区设置

docker log管理和时区设置

1:docker log

1.1 docker log Description

  • docker 的日志分为两种,第一种是 docker 引擎的日志,第二种是 docker 容器运行时的服务日志,默认为 json-file 格式;
  • json-file 日志驱动 记录从容器的 STOUT/STDERR 的输出 ,用 JSON 的格式写到文件中,日志中不仅包含着 输出日志,还有时间戳和 输出格式.
  • docker 所有的日志驱动如下:
    日志驱动descriptionothers
    none运行的容器没有日志,docker logs 也不返回任何输出
    local日志以自定义格式存储,旨在实现最小开销
    json-file日志格式为 JSON,Docker 的默认日志记录驱动程序
    syslog将日志消息写入 syslog。该 syslog 守护程序必须在主机上运行
    journald将日志消息写入 journald。该 journald 守护程序必须在主机上运行
    gelf将日志消息写入 Graylog 扩展日志格式(GELF)端点,例如 Graylog 或 Logstash
    fluentd将日志消息写入 fluentd(转发输入)。该 fluentd 守护程序必须在主机上运行
    awslogs将日志消息写入 Amazon CloudWatch Logs
    splunk使用 HTTP 事件收集器将日志消息写入 splunk
    etwlogs将日志消息写为 Windows 事件跟踪(ETW)事件。仅适用于 Windows 平台
    gcplogs将日志消息写入 Google Cloud Platform(GCP)Logging
    logentries将日志消息写入 Rapid7 Logentries
  • 常用的驱动是 json-filesyslog ,journald 这三种,json-file 是默认的日志驱动;
  • docker 的日志通常默认是保存在 /var/lib/docker/containers/容器 ID/ 目录下,日志文件名称为 容器 ID-json-log,当然,docker 日志也有级别,通常级别为 info。

1.2 磁盘占用分析

1.2.1 df -ah 查看系统中文件的使用情况

  • df (disk free) 检查文件系统占用磁盘情况;
1
2
3
4
5
Size 分割区总容量
Used 已使用的大小
Avail 剩下的大小
Use% 使用的百分比
Mounted on 路径地址

1.2.2 du -sh 查看当前目录下各个文件及目录占用空间大小

  • du (disk usage) 检查目录占用磁盘情况;
1
2
du -h --max-depth=1 /root/* 查看目录下的所有文件大小
du -h --max-depth=1 /root 列出root目录下面所有的一级目录文件大小;

1.2.3 df -h 和 du -sh 显示的磁盘大小不一致原因及解决办法

  • 原因 du 是根据文件名进行的空间统计,使用 rm 时该文件对系统来说已经不可见,所以不会统计这个文件。 df 则是磁盘实际占用的数量
  • du -h /var/lib/docker/
  • df -h /var/lib/docker/
  • 解决办法
    • (1/2) lsof 查看使用文件的进程,结束进程。
      • lsof | grep test.log
      • ps aux | grep 28600
      • kill -9 28600
    • (2/2) 或使用清空的方式代替 rm,如 echo > /tmp/test.log
      • 日志文件清空一般不是删除 rm 命令,而是 > 文件名这样的清空方式.

1.3 docker log 管理

  • 日志文件清空一般不是删除 rm 命令,而是 > 文件名这样的清空方式;

01-go使用Linux内核端口复用进行LoadBalance

1 查看 Linux Kernal

  • socket 五元组 {<protocol>, <src addr>, <src port>, <dest addr>, <dest port>}
  • uname -a
  • Linux 3.9 内核引入了 SO_REUSEPORT选项,可支持多个进程或者线程绑定到同一个端口,用于提高服务器的性能。他的特性包含一下几点:
    • 允许多个 socket 套接字 bind 同一个 TCP/UDP 端口;
      • 每一个线程拥有自己的服务器套接字;
      • 在服务器套接字上没有了锁的竞争;
    • 内核层面实现负载均衡;
    • 安全层面,listen 同一个端口的 socket 套接字只能位于同一个 user 下。
  • 有了SO_RESUEPORT后,每个进程可以 bind 相同的地址和端口,各自是独立平等的;
  • 让多进程监听同一个端口,各个进程中 accept socket fd 不一样,有新连接建立时,内核只会调度一个进程来 accept,并且保证调度的均衡性。

端口复用图示

01-grpc安装(cpp和go)

1 c++ 配置 GRPC

https://grpc.io/docs/languages/cpp/quickstart/

1.1 setup dir

1
2
3
4
5
export MY_INSTALL_DIR=$HOME/.local

mkdir -p $MY_INSTALL_DIR

export PATH="$MY_INSTALL_DIR/bin:$PATH"

1.2 install cmake

1
sudo apt install -y cmake

1.3 Install other required tools

1
sudo apt install -y build-essential autoconf libtool pkg-config

1.4 git clone grpc repo

1
git clone --recurse-submodules -b v1.41.0 https://github.com/grpc/grpc

1.5 Build and install gRPC and Protocol Buffers

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
cd grpc
mkdir -p cmake/build
pushd cmake/build
cmake -DgRPC_INSTALL=ON \
    -DgRPC_BUILD_TESTS=OFF \
    -DCMAKE_INSTALL_PREFIX=$MY_INSTALL_DIR \
    ../..
make -j
make install
popd
1
2
3
4
5
# cmake
mkdir -p cmake/build
cd cmake/build
cmake ../..
make

1.6 编写 proto 文件

1.7 编写 server/client 文件

1.8 执行

2 golang grpc 环境搭建

https://grpc.io/docs/languages/go/quickstart/

011-docker部署gitlab

docker部署gitlab

1:gitlab docker 镜像 pull

  • docker pull gitlab/gitlab-ce:latest
  • 新建目录~/gitlab

2:启动脚本

~/gitlab/start.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
#!/bin/bash
GITLAB_DIR=/home/lucas/gitlab/data

docker run -d \
    -p 5443:443 \
    -p 9998:9998 \
    -p 2222:22 \
    --name gitlab \
    -v ${GITLAB_DIR}/config:/etc/gitlab \
    -v ${GITLAB_DIR}/logs:/var/log/gitlab \
    -v ${GITLAB_DIR}/data:/var/opt/gitlab \
    gitlab/gitlab-ce:latest

3:修改gitlab.rb

~/gitlab/data/config/gitlab.rb

01-note笔记

note笔记

1:note 文件

1.1 alias 别名设置

1
2
3
4
alias gs = "git status"
alias ga = "git add -A"
alias gd = "git diff"
alias gc = "git commit -m"

1.2 git 局部代理

1
git clone -c http.proxy="127.0.0.1:7078" xxx

1.3 golang 交叉编译

1
2
3
4
5
6
7
8
CGO_ENABLED=0 GOOS=linux GOARCH=amd64    go build -v -a main.go

CGO_ENABLED=0 GOOS=darwin GOARCH=amd64   go build -v -a main.go

CGO_ENABLED=0 GOOS=windows GOARCH=amd64  go build -v -a main.go

# 编译时添加参数
 -ldflags "-X 'main.Version=v1.0.0' -X 'main.BuildTime=`date +"%Y-%m-%d %H:%M:%S"`' -X 'main.GoVersion=`go version`' "

1.4 date 和时间戳转换

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# linux  date和时间戳转换
1:查看当前时间
date +"%Y-%m-%d %H:%M:%S"

输出:"2021-03-25 18:12:38"

2:查看指定时间,转为时间戳
date +%s
date -d "2021-03-25 18:12:38" +"%Y-%m-%d %H:%M:%S"
date -d "2021-03-25 18:12:38" +%s

3:时间戳转换为date
date -d @1416387827
date -d @1416387827 +"%Y-%m-%d %H:%M:%S"

1.5 minikube start

1
2
3
4
5
6
minikube start \
    --driver docker \
    --image-mirror-country=cn \
    --registry-mirror https://3laho3y3.mirror.aliyuncs.com \
    --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \
    --kubernetes-version=v1.20.7

1.6 ip 相关

1
2
3
4
5
6
7
#
curl ipinfo.io
#
curl -vv www.google.com

# ------------------查看ip--------------------
ip addr show | grep "inet"

1.7 查看 Linux 详细 info

1
cat /etc/*elease

1.8 WSL2 压缩磁盘空间

1
2
3
4
# ----------------------WSL2压缩磁盘空间------------------------
PowerShell,直接运行 diskpart 命令就好
select vdisk file=D:\wsl\Ubuntu-20.04\ext4.vhdx
compact vdisk

01-top命令基本使用

1: Top 命令按键

options按键说明示例其他
-u显示特定 user 的进程top -u lucas
-p显示特定 PID 的进程top -p 10001 10002
-n10 次输出后退出程top -n 10
1按数字 1 可以显示每个 CPU 核心使用情况
M根据驻留内存RES大小进行排序,由大到小(注意大小写)
RRES反序排序
P根据 CPU 使用百分比大小进行
T根据时间/累计时间进行排序
c切换显示命令名称和完整命令行
i忽略闲置和僵死进程。这是一个开关式命令。
f更改显示内容通过 f 键可以选择显示的内容。按 f 键之后会显示列的列表,空格确定显示或者取消显示 a 是切换
d更改刷新时间
o进入 top 后,按下 o 后,会让你输入过滤条件,输入 COMMAND=nginx,现在就只剩下模糊匹配 nginx 关键字的进程了
Shift+P对进程 CPU 利用率进行排序
Shift+O对进程 Memory 利用率进行排序

2 Top 参数说明

2.1 CPU 状态信息

0.0%us 【user space】— 用户空间占用 CPU 的百分比。

01-zshrc 笔记

zshrc笔记

1:zshrc 文件

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
# If you come from bash you might have to change your $PATH.
# export PATH=$HOME/bin:/usr/local/bin:$PATH

# Path to your oh-my-zsh installation.
export ZSH="/home/lucas/.oh-my-zsh"

# Set name of the theme to load --- if set to "random", it will
# load a random theme each time oh-my-zsh is loaded, in which case,
# to know which specific one was loaded, run: echo $RANDOM_THEME
# See https://github.com/ohmyzsh/ohmyzsh/wiki/Themes

# 修改默认主题
ZSH_THEME="ys"

# Set list of themes to pick from when loading at random
# Setting this variable when ZSH_THEME=random will cause zsh to load
# a theme from this variable instead of looking in $ZSH/themes/
# If set to an empty array, this variable will have no effect.
# ZSH_THEME_RANDOM_CANDIDATES=( "robbyrussell" "agnoster" )

# Uncomment the following line to use case-sensitive completion.
# CASE_SENSITIVE="true"

# Uncomment the following line to use hyphen-insensitive completion.
# Case-sensitive completion must be off. _ and - will be interchangeable.
# HYPHEN_INSENSITIVE="true"

# Uncomment the following line to disable bi-weekly auto-update checks.
# DISABLE_AUTO_UPDATE="true"

# Uncomment the following line to automatically update without prompting.
# DISABLE_UPDATE_PROMPT="true"

# Uncomment the following line to change how often to auto-update (in days).
# export UPDATE_ZSH_DAYS=13

# Uncomment the following line if pasting URLs and other text is messed up.
# DISABLE_MAGIC_FUNCTIONS=true

# Uncomment the following line to disable colors in ls.
# DISABLE_LS_COLORS="true"

# Uncomment the following line to disable auto-setting terminal title.
# DISABLE_AUTO_TITLE="true"

# Uncomment the following line to enable command auto-correction.
# ENABLE_CORRECTION="true"

# Uncomment the following line to display red dots whilst waiting for completion.
# COMPLETION_WAITING_DOTS="true"

# Uncomment the following line if you want to disable marking untracked files
# under VCS as dirty. This makes repository status check for large repositories
# much, much faster.
# DISABLE_UNTRACKED_FILES_DIRTY="true"

# Uncomment the following line if you want to change the command execution time
# stamp shown in the history command output.
# You can set one of the optional three formats:
# "mm/dd/yyyy"|"dd.mm.yyyy"|"yyyy-mm-dd"
# or set a custom format using the strftime function format specifications,
# see 'man strftime' for details.
# HIST_STAMPS="mm/dd/yyyy"

# Would you like to use another custom folder than $ZSH/custom?
# ZSH_CUSTOM=/path/to/new-custom-folder

# Which plugins would you like to load?
# Standard plugins can be found in $ZSH/plugins/
# Custom plugins may be added to $ZSH_CUSTOM/plugins/
# Example format: plugins=(rails git textmate ruby lighthouse)
# Add wisely, as too many plugins slow down shell startup.
plugins=(
    git
    zsh-syntax-highlighting
    zsh-autosuggestions
    sudo
)

source $ZSH/oh-my-zsh.sh

# User configuration

# export MANPATH="/usr/local/man:$MANPATH"

# You may need to manually set your language environment
# export LANG=en_US.UTF-8

# Preferred editor for local and remote sessions
# if [[ -n $SSH_CONNECTION ]]; then
#   export EDITOR='vim'
# else
#   export EDITOR='mvim'
# fi

# Compilation flags
# export ARCHFLAGS="-arch x86_64"

# Set personal aliases, overriding those provided by oh-my-zsh libs,
# plugins, and themes. Aliases can be placed here, though oh-my-zsh
# users are encouraged to define aliases within the ZSH_CUSTOM folder.
# For a full list of active aliases, run `alias`.
#
# Example aliases
# alias zshconfig="mate ~/.zshrc"
# alias ohmyzsh="mate ~/.oh-my-zsh"

source $ZSH_CUSTOM/plugins/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
source $ZSH_CUSTOM/plugins/zsh-autosuggestions/zsh-autosuggestions.zsh

# 修改默认编辑器为vim
export EDITOR=/usr/bin/vim

export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export GOBIN=$GOPATH/bin
export PATH=$GOPATH:$GOBIN:$GOROOT/bin:$PATH

export RUSTUP_DIST_SERVER=https://mirrors.ustc.edu.cn/rust-static
export RUSTUP_UPDATE_ROOT=https://mirrors.ustc.edu.cn/rust-static/rustup

export PATH="$HOME/bin:$HOME/.local/bin:$PATH"
export PATH="$PATH:/mnt/c/Program\ Files/Docker/Docker/resources/bin"
export PATH="$PATH:/usr/local/lib/node-v16.4.2-linux-x64/bin"

# export DOCKER_HOST="tcp://0.0.0.0:2375"

# alias
# alias docker=docker.exe
# alias docker-compose=docker-compose.exe
# alias go colorgo

# history 命令显示时间
export HIST_STAMPS="yyyy-mm-dd"

# ----------------proxy----------------------
#
WindowsIP=$(cat /etc/resolv.conf | grep -oP '(?<=nameserver\ ).*')
PROXY_PORT=7078
PROXY_HTTP="http://${WindowsIP}:${PROXY_PORT}"
# Git & SSH for Git proxy
proxy_git() {
    git config --global http.https://github.com.proxy ${PROXY_HTTP}
    if ! grep -qF "Host github.com" ~/.ssh/config; then
        echo "Host github.com" >>~/.ssh/config
        echo "  User git" >>~/.ssh/config
        echo "  ProxyCommand nc -X 5 -x ${PROXY_SOCKS5} %h %p" >>~/.ssh/config
    else
        lino=$(($(awk '/Host github.com/{print NR}' ~/.ssh/config) + 2))
        sed -i "${lino}c\  ProxyCommand nc -X 5 -x ${PROXY_SOCKS5} %h %p" ~/.ssh/config
    fi
}
# Set proxy
set_proxy() {
    export http_proxy="${PROXY_HTTP}"
    export https_proxy="${PROXY_HTTP}"
    proxy_git
}
# Unset proxy
unset_proxy() {
    unset http_proxy
    unset https_proxy
    git config --global --unset http.https://github.com.proxy
}

# Set alias
alias proxy=set_proxy
alias deproxy=unset_proxy

alias px='proxychains4' # 此后 px curl google.com

# wsl sync time
# alias synctime='sudo ntpdate asia.pool.ntp.org;sudo ntpdate time.nist.gov;'
alias synctime1='sudo ntpdate asia.pool.ntp.org;'
alias synctime2='sudo ntpdate time.nist.gov;'

# 手动释放内存
alias freemem='~/myshell/free-mem.sh'

# alias godoc='GO111MODULE=off godoc'
alias fy='unproxy dict'

2:其他说明

01-Golang Files and I/O

Golang Files and I/O

Table of Contents

一:读文件

1.1 读一个 whole 文件

1
2
3
4
5
d, err := ioutil.ReadFile("foo.txt")
if err != nil {
    log.Fatalf("ioutil.ReadFile failed with '%s'\n", err)
}
fmt.Printf("Size of 'foo.txt': %d bytes\n", len(d))

1.2 读一个文件,close

1
2
3
4
5
f, err := os.Open("foo.txt")
if err != nil {
    log.Fatalf("os.Open failed with '%s'\n", err)
}
defer f.Close()

1.3 read file by line

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// ReadLines reads all lines from a file
func ReadLines(filePath string) ([]string, error) {
	file, err := os.OpenFile(filePath, os.O_RDONLY, 0644)
	if err != nil {
		return nil, err
	}
	defer file.Close()
	scanner := bufio.NewScanner(file)
	res := make([]string, 0)
	for scanner.Scan() {
		line := scanner.Text()
		res = append(res, line)
	}
	if err = scanner.Err(); err != nil {
		return nil, err
	}
	return res, nil
}

func main() {
	path := "main.go"
	lines, err := ReadLines(path)
	if err != nil {
		log.Fatalf("ReadLines failed with '%s'\n", err)
	}
	fmt.Printf("File %s has %d lines\n", path, len(lines))
}

二:写文件

2.1 Write the whole file

1
2
3
4
5
d := []byte("content of the file")
err := ioutil.WriteFile("foo.txt", d, 0644)
if err != nil {
    log.Fatalf("ioutil.WriteFile failed with '%s'\n", err)
}

2.2 Open file for writing

1
2
3
4
f, err := os.Create("foo.txt")
if err != nil {
    log.Fatalf("os.Open failed with '%s'\n", err)
}

2.3 Open file for appending

1
2
3
4
f, err := os.OpenFile(filePath, os.O_WRONLY | os.O_APPEND | os.O_CREATE, 0666)
if err != nil {
    log.Fatalf("os.Open failed with '%s'\n", err)
}

2.4 Write to a file

1
2
d := []byte("data to write")
nWritten, err := f.Write(d)

2.5 File permissions when creating files

使用 os.Create 或创建新文件时 os.OpenFile,需要为新文件提供文件权限。

01-微服务限流简单实现

微服务限流简单实现

Table of Contents

一:限流的目的

保障服务稳定的三大利器:熔断、降级、服务限流。今天和大家谈谈限流算法的几种实现方式,本文所说的限流并非是 Nginx、网关层面的限流,而是业务代码中的逻辑限流。