[Python] CPU spike occurs when uploading data via SFTP or Web UI (upload local file or through...

Discussão em 'Python' iniciado por Stack, Fevereiro 12, 2025.

  1. Stack

    Stack Membro Participativo

    I expect very low CPU usage when uploading data via SFTP or Web UI in MinIO. But CPU spike occurs when uploading data via SFTP or Web UI.

    • I am uploading multiple image data over SFTP/Web using python asyncssh package in MinIO. (Data Lake)
    • Both sftp.mput and sftp.put functions in the asyncssh package cause CPU spikes.
    • Even if an upload delay is given, a CPU spike occurs at the moment of upload.
    • I also checked SSE and compression, but they are disabled.
      ubuntu@mycom:~$ mc admin config get admin compression
      compression enable=off allow_encryption=off extensions=.txt,.log,.csv,.json,.tar,.xml,.bin mime_types=text/*,application/json,application/xml,binary/octet-stream

      ubuntu@mycom:~$ mc admin config get admin kms
      mc: <ERROR> Unable to get server '[kms]' config: unknown subsystem: kms.

    The problem is that the CPU usage goes up drastically. If there are 10 sftp connections, the server dies.

    server spec:

    • EC2 instance: m7i-xlarge
    • 4core
    • 16GBi Mem
    • EBS type single ssd drive

    (htop screen when 5 sftp connections are made) [​IMG]

    (docker stats screen when 5 sftp connections are made) [​IMG]


    I ran the minio service with the docker-compose.yml information below.

    minio:
    image: minio/minio:latest
    container_name: myminio_dev
    restart: always
    environment:
    MINIO_ROOT_USER: ${MINIO_ROOT_USERNAME}
    MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
    MINIO_PORT: ${MINIO_PORT}
    MINIO_CONSOLE_PORT: ${MINIO_CONSOLE_PORT}
    MINIO_SFTP_PORT: ${MINIO_SFTP_PORT}
    command: server /data --console-address ":${MINIO_CONSOLE_PORT}" --sftp "address=:${MINIO_SFTP_PORT}" --sftp "ssh-private-key=${MINIO_SSH_PRIVATE_KEY}" --sftp "trusted-user-ca-key=${MINIO_TRUSTED_USER_CA_KEY}"
    ports:
    - "${MINIO_PORT}:${MINIO_PORT}" # S3 API access port
    - "${MINIO_CONSOLE_PORT}:${MINIO_CONSOLE_PORT}" # MinIO console access port
    - "${MINIO_FTP_PORT}:${MINIO_FTP_PORT}" # MinIO FTP connection port
    - "${MINIO_FTP_PASV_PORT}:${MINIO_FTP_PASV_PORT}" # MinIO FTP Data ports
    - "${MINIO_SFTP_PORT}:${MINIO_SFTP_PORT}" # MinIO SFTP connection port
    volumes:
    - /mnt/minio:/data
    - ${MINIO_CERT_PATH}:${MINIO_CERT_PATH}:ro

    • mc admin trace --verbose myminio

    127.0.0.1:9000 [REQUEST s3.GetBucketLocation] [2025-02-12T02:21:27.719] [Client IP: 111.111.111.111]
    127.0.0.1:9000 GET /device/?location=
    127.0.0.1:9000 Proto: HTTP/1.1
    127.0.0.1:9000 Host: 127.0.0.1:9000
    127.0.0.1:9000 Authorization: AWS4-HMAC-SHA256 Credential=6d60e92a-3e54-4ced-879c-e93423619d32/20250212/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7d41dce67a4751f47c5d0599480ec32c41dbff66d0d8fe691915d2466929e2da
    127.0.0.1:9000 Content-Length: 0
    127.0.0.1:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.83
    127.0.0.1:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    127.0.0.1:9000 X-Amz-Date: 20250212T022127Z
    127.0.0.1:9000 X-Forwarded-For: 111.111.111.111
    127.0.0.1:9000
    127.0.0.1:9000 [RESPONSE] [2025-02-12T02:21:27.721] [ Duration 2.03ms TTFB 2.013522ms ↑ 93 B ↓ 142 B ]
    127.0.0.1:9000 200 OK
    127.0.0.1:9000 X-Ratelimit-Remaining: 3283
    127.0.0.1:9000 X-Xss-Protection: 1; mode=block
    127.0.0.1:9000 Content-Type: application/xml
    127.0.0.1:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
    127.0.0.1:9000 Vary: Origin,Accept-Encoding
    127.0.0.1:9000 X-Amz-Bucket-Region: ap-northeast-2
    127.0.0.1:9000 X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
    127.0.0.1:9000 X-Ratelimit-Limit: 3283
    127.0.0.1:9000 Accept-Ranges: bytes
    127.0.0.1:9000 Content-Length: 142
    127.0.0.1:9000 Server: MinIO
    127.0.0.1:9000 X-Amz-Request-Id: 18235496CACB8D29
    127.0.0.1:9000 X-Content-Type-Options: nosniff
    127.0.0.1:9000 <?xml version="1.0" encoding="UTF-8"?>
    <LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">ap-northeast-2</LocationConstraint>
    127.0.0.1:9000
    127.0.0.1:9000 [REQUEST s3.HeadObject] [2025-02-12T02:21:27.721] [Client IP: 111.111.111.111]
    127.0.0.1:9000 HEAD /device/6d60e92a-3e54-4ced-879c-e93423619d32
    127.0.0.1:9000 Proto: HTTP/1.1
    127.0.0.1:9000 Host: 127.0.0.1:9000
    127.0.0.1:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    127.0.0.1:9000 X-Amz-Date: 20250212T022127Z
    127.0.0.1:9000 X-Forwarded-For: 111.111.111.111
    127.0.0.1:9000 Authorization: AWS4-HMAC-SHA256 Credential=6d60e92a-3e54-4ced-879c-e93423619d32/20250212/ap-northeast-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9b6a05f795e981fe19f8f9378a8b25a8ec23fae15e05dde681b23eed15798b5c
    127.0.0.1:9000 Content-Length: 0
    127.0.0.1:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.83
    127.0.0.1:9000
    127.0.0.1:9000 [RESPONSE] [2025-02-12T02:21:27.721] [ Duration 279µs TTFB 254.422µs ↑ 93 B ↓ 0 B ]
    127.0.0.1:9000 404 Not Found
    127.0.0.1:9000 X-Amz-Bucket-Region: ap-northeast-2
    127.0.0.1:9000 X-Minio-Error-Desc: "The specified key does not exist."
    127.0.0.1:9000 X-Ratelimit-Remaining: 3283
    127.0.0.1:9000 Vary: Origin,Accept-Encoding
    127.0.0.1:9000 X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
    127.0.0.1:9000 X-Content-Type-Options: nosniff
    127.0.0.1:9000 X-Minio-Error-Code: NoSuchKey
    127.0.0.1:9000 Server: MinIO
    127.0.0.1:9000 X-Amz-Request-Id: 18235496CAEF1FB2
    127.0.0.1:9000 X-Ratelimit-Limit: 3283
    127.0.0.1:9000 X-Xss-Protection: 1; mode=block
    127.0.0.1:9000 Accept-Ranges: bytes
    127.0.0.1:9000 Content-Length: 0
    127.0.0.1:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
    127.0.0.1:9000
    127.0.0.1:9000
    127.0.0.1:9000 [REQUEST s3.GetBucketLocation] [2025-02-12T02:21:27.755] [Client IP: 111.111.111.111]
    127.0.0.1:9000 GET /device/?location=
    127.0.0.1:9000 Proto: HTTP/1.1
    127.0.0.1:9000 Host: 127.0.0.1:9000
    127.0.0.1:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.83
    127.0.0.1:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    127.0.0.1:9000 X-Amz-Date: 20250212T022127Z
    127.0.0.1:9000 X-Forwarded-For: 111.111.111.111
    127.0.0.1:9000 Authorization: AWS4-HMAC-SHA256 Credential=6d60e92a-3e54-4ced-879c-e93423619d32/20250212/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7d41dce67a4751f47c5d0599480ec32c41dbff66d0d8fe691915d2466929e2da
    127.0.0.1:9000 Content-Length: 0
    127.0.0.1:9000
    127.0.0.1:9000 [RESPONSE] [2025-02-12T02:21:27.756] [ Duration 193µs TTFB 182.706µs ↑ 93 B ↓ 142 B ]
    127.0.0.1:9000 200 OK
    127.0.0.1:9000 X-Ratelimit-Remaining: 3283
    127.0.0.1:9000 X-Xss-Protection: 1; mode=block
    127.0.0.1:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
    127.0.0.1:9000 Vary: Origin,Accept-Encoding
    127.0.0.1:9000 X-Amz-Bucket-Region: ap-northeast-2
    127.0.0.1:9000 X-Content-Type-Options: nosniff
    127.0.0.1:9000 X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
    127.0.0.1:9000 X-Amz-Request-Id: 18235496CCF9E232
    127.0.0.1:9000 X-Ratelimit-Limit: 3283
    127.0.0.1:9000 Accept-Ranges: bytes
    127.0.0.1:9000 Content-Length: 142
    127.0.0.1:9000 Content-Type: application/xml
    127.0.0.1:9000 Server: MinIO
    127.0.0.1:9000 <?xml version="1.0" encoding="UTF-8"?>
    <LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">ap-northeast-2</LocationConstraint>
    127.0.0.1:9000
    127.0.0.1:9000 [REQUEST s3.HeadBucket] [2025-02-12T02:21:27.756] [Client IP: 111.111.111.111]
    127.0.0.1:9000 HEAD /device/
    127.0.0.1:9000 Proto: HTTP/1.1
    127.0.0.1:9000 Host: 127.0.0.1:9000
    127.0.0.1:9000 Authorization: AWS4-HMAC-SHA256 Credential=6d60e92a-3e54-4ced-879c-e93423619d32/20250212/ap-northeast-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ab3abf07a2b4380e10aa2fefad10f2fcec67b77fecfacfbfc5c0086ce70c37a3
    127.0.0.1:9000 Content-Length: 0
    127.0.0.1:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.83
    127.0.0.1:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    127.0.0.1:9000 X-Amz-Date: 20250212T022127Z
    127.0.0.1:9000 X-Forwarded-For: 111.111.111.111
    127.0.0.1:9000
    127.0.0.1:9000 [RESPONSE] [2025-02-12T02:21:27.756] [ Duration 346µs TTFB 303.346µs ↑ 93 B ↓ 0 B ]
    127.0.0.1:9000 200 OK
    127.0.0.1:9000 X-Amz-Request-Id: 18235496CD02C67A
    127.0.0.1:9000 X-Content-Type-Options: nosniff
    127.0.0.1:9000 X-Ratelimit-Remaining: 3283
    127.0.0.1:9000 X-Xss-Protection: 1; mode=block
    127.0.0.1:9000 Content-Type: application/xml
    127.0.0.1:9000 Server: MinIO
    127.0.0.1:9000 Vary: Origin,Accept-Encoding
    127.0.0.1:9000 X-Amz-Bucket-Region: ap-northeast-2
    127.0.0.1:9000 X-Ratelimit-Limit: 3283
    127.0.0.1:9000 Accept-Ranges: bytes
    127.0.0.1:9000 Content-Length: 0
    127.0.0.1:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
    127.0.0.1:9000 X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
    127.0.0.1:9000
    127.0.0.1:9000
    127.0.0.1:9000 [REQUEST s3.NewMultipartUpload] [2025-02-12T02:21:27.757] [Client IP: 111.111.111.111]
    127.0.0.1:9000 POST /device/6d60e92a-3e54-4ced-879c-e93423619d32/GBR_NorthEastLive_GHCBSA1-01_20240909T121108%2B0000_15047.jpg?uploads=
    127.0.0.1:9000 Proto: HTTP/1.1
    127.0.0.1:9000 Host: 127.0.0.1:9000
    127.0.0.1:9000 Content-Length: 0
    127.0.0.1:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.83
    127.0.0.1:9000 X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
    127.0.0.1:9000 X-Forwarded-For: 111.111.111.111
    127.0.0.1:9000 Authorization: AWS4-HMAC-SHA256 Credential=6d60e92a-3e54-4ced-879c-e93423619d32/20250212/ap-northeast-2/s3/aws4_request, SignedHeaders=content-type;host;x-amz-checksum-algorithm;x-amz-content-sha256;x-amz-date, Signature=805c0ee0e95dec65809396cd95687b7a24dc7f30d5460345b71c2231a8226ba0
    127.0.0.1:9000 Content-Type: image/jpeg
    127.0.0.1:9000 X-Amz-Checksum-Algorithm: CRC32C
    127.0.0.1:9000 X-Amz-Date: 20250212T022127Z
    127.0.0.1:9000
    127.0.0.1:9000 [RESPONSE] [2025-02-12T02:21:27.763] [ Duration 6.011ms TTFB 5.997517ms ↑ 131 B ↓ 425 B ]
    127.0.0.1:9000 200 OK
    127.0.0.1:9000 X-Amz-Request-Id: 18235496CD102674
    127.0.0.1:9000 X-Ratelimit-Limit: 3283
    127.0.0.1:9000 X-Ratelimit-Remaining: 3283
    127.0.0.1:9000 X-Amz-Bucket-Region: ap-northeast-2
    127.0.0.1:9000 Server: MinIO
    127.0.0.1:9000 X-Amz-Checksum-Algorithm: CRC32C
    127.0.0.1:9000 Accept-Ranges: bytes
    127.0.0.1:9000 Vary: Origin,Accept-Encoding
    127.0.0.1:9000 X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
    127.0.0.1:9000 X-Xss-Protection: 1; mode=block
    127.0.0.1:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
    127.0.0.1:9000 Content-Type: application/xml
    127.0.0.1:9000 X-Content-Type-Options: nosniff
    127.0.0.1:9000 Content-Length: 425
    127.0.0.1:9000 <?xml version="1.0" encoding="UTF-8"?>
    <InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>device</Bucket><Key>6d60e92a-3e54-4ced-879c-e93423619d32/GBR_NorthEastLive_GHCBSA1-01_20240909T121108+0000_15047.jpg</Key><UploadId>NWQ2ZmM0MWItY2JiNi00MTlmLWE2MGQtNjc1ZWZlODNjMDZkLjU2Y2NlMjc5LWEzNzYtNGI2My1iZWJiLWVhYjNhYjM3ZmI2ZngxNzM5MzI2ODg3NzU3NjEzOTY4</UploadId></InitiateMultipartUploadResult>
    127.0.0.1:9000
    127.0.0.1:9000 [REQUEST s3.PutObjectPart] [2025-02-12T02:21:27.825] [Client IP: 111.111.111.111]
    127.0.0.1:9000 PUT /device/6d60e92a-3e54-4ced-879c-e93423619d32/GBR_NorthEastLive_GHCBSA1-01_20240909T121108%2B0000_15047.jpg?partNumber=1&uploadId=NWQ2ZmM0MWItY2JiNi00MTlmLWE2MGQtNjc1ZWZlODNjMDZkLjU2Y2NlMjc5LWEzNzYtNGI2My1iZWJiLWVhYjNhYjM3ZmI2ZngxNzM5MzI2ODg3NzU3NjEzOTY4
    127.0.0.1:9000 Proto: HTTP/1.1
    127.0.0.1:9000 Host: 127.0.0.1:9000
    127.0.0.1:9000 X-Forwarded-For: 111.111.111.111
    127.0.0.1:9000 Authorization: AWS4-HMAC-SHA256 Credential=6d60e92a-3e54-4ced-879c-e93423619d32/20250212/ap-northeast-2/s3/aws4_request, SignedHeaders=host;x-amz-checksum-crc32c;x-amz-content-sha256;x-amz-date, Signature=a0b1916ff74f5d55be19efdc040d1c714240cce329bc903f853cc191434bbf87
    127.0.0.1:9000 Content-Length: 63971
    127.0.0.1:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.83
    127.0.0.1:9000 X-Amz-Checksum-Crc32c: soz+Pw==
    127.0.0.1:9000 X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
    127.0.0.1:9000 X-Amz-Date: 20250212T022127Z
    127.0.0.1:9000 <BLOB>
    127.0.0.1:9000 [RESPONSE] [2025-02-12T02:21:27.835] [ Duration 9.563ms TTFB 9.532614ms ↑ 63 KiB ↓ 0 B ]
    127.0.0.1:9000 200 OK
    127.0.0.1:9000 X-Amz-Bucket-Region: ap-northeast-2
    127.0.0.1:9000 X-Amz-Request-Id: 18235496D125361C
    127.0.0.1:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
    127.0.0.1:9000 X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
    127.0.0.1:9000 X-Ratelimit-Limit: 3283
    127.0.0.1:9000 X-Ratelimit-Remaining: 3283
    127.0.0.1:9000 X-Xss-Protection: 1; mode=block
    127.0.0.1:9000 ETag: "73772167823c98517f206e8516eee282"
    127.0.0.1:9000 Vary: Origin,Accept-Encoding
    127.0.0.1:9000 X-Amz-Checksum-Crc32c: soz+Pw==
    127.0.0.1:9000 Accept-Ranges: bytes
    127.0.0.1:9000 Content-Length: 0
    127.0.0.1:9000 Server: MinIO
    127.0.0.1:9000 X-Content-Type-Options: nosniff
    127.0.0.1:9000 <BLOB>
    127.0.0.1:9000
    127.0.0.1:9000 [REQUEST s3.CompleteMultipartUpload] [2025-02-12T02:21:27.836] [Client IP: 111.111.111.111]
    127.0.0.1:9000 POST /device/6d60e92a-3e54-4ced-879c-e93423619d32/GBR_NorthEastLive_GHCBSA1-01_20240909T121108%2B0000_15047.jpg?uploadId=NWQ2ZmM0MWItY2JiNi00MTlmLWE2MGQtNjc1ZWZlODNjMDZkLjU2Y2NlMjc5LWEzNzYtNGI2My1iZWJiLWVhYjNhYjM3ZmI2ZngxNzM5MzI2ODg3NzU3NjEzOTY4
    127.0.0.1:9000 Proto: HTTP/1.1
    127.0.0.1:9000 Host: 127.0.0.1:9000
    127.0.0.1:9000 Authorization: AWS4-HMAC-SHA256 Credential=6d60e92a-3e54-4ced-879c-e93423619d32/20250212/ap-northeast-2/s3/aws4_request, SignedHeaders=content-type;host;x-amz-checksum-crc32c;x-amz-content-sha256;x-amz-date, Signature=f6a3f50afac7da32754cad037fa555cdd82af9989cdf3b2e6a5b55fea5291545
    127.0.0.1:9000 Content-Length: 224
    127.0.0.1:9000 Content-Type: application/octet-stream
    127.0.0.1:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.83
    127.0.0.1:9000 X-Amz-Checksum-Crc32c: aPggMg==
    127.0.0.1:9000 X-Amz-Content-Sha256: c268af49ab2e6b9d4335e8dd3ac574f6642f3c5169b994f741b5740c814c8183
    127.0.0.1:9000 X-Amz-Date: 20250212T022127Z
    127.0.0.1:9000 X-Forwarded-For: 111.111.111.111
    127.0.0.1:9000 <CompleteMultipartUpload xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Part><PartNumber>1</PartNumber><ETag>73772167823c98517f206e8516eee282</ETag><ChecksumCRC32C>soz+Pw==</ChecksumCRC32C></Part></CompleteMultipartUpload>
    127.0.0.1:9000 [RESPONSE] [2025-02-12T02:21:27.840] [ Duration 4.309ms TTFB 4.2764ms ↑ 352 B ↓ 526 B ]
    127.0.0.1:9000 200 OK
    127.0.0.1:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
    127.0.0.1:9000 Vary: Origin,Accept-Encoding
    127.0.0.1:9000 X-Amz-Request-Id: 18235496D1C12C9C
    127.0.0.1:9000 X-Xss-Protection: 1; mode=block
    127.0.0.1:9000 ETag: "88128f2e7759186315b86c0b1bef696b-1"
    127.0.0.1:9000 x-amz-version-id: b56e6fdc-0881-4bc6-a89c-d8a82c63a3cd
    127.0.0.1:9000 Content-Type: application/xml
    127.0.0.1:9000 X-Amz-Checksum-Crc32c: aPggMg==-1
    127.0.0.1:9000 X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
    127.0.0.1:9000 X-Content-Type-Options: nosniff
    127.0.0.1:9000 X-Ratelimit-Limit: 3283
    127.0.0.1:9000 Accept-Ranges: bytes
    127.0.0.1:9000 Content-Length: 526
    127.0.0.1:9000 Server: MinIO
    127.0.0.1:9000 X-Amz-Bucket-Region: ap-northeast-2
    127.0.0.1:9000 X-Ratelimit-Remaining: 3283
    127.0.0.1:9000 <?xml version="1.0" encoding="UTF-8"?>
    <CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>http://127.0.0.1:9000/device/6d60e92a-3e54-4ced-879c-e93423619d32/GBR_NorthEastLive_GHCBSA1-01_20240909T121108+0000_15047.jpg</Location><Bucket>device</Bucket><Key>6d60e92a-3e54-4ced-879c-e93423619d32/GBR_NorthEastLive_GHCBSA1-01_20240909T121108+0000_15047.jpg</Key><ETag>&#34;88128f2e7759186315b86c0b1bef696b-1&#34;</ETag><ChecksumCRC32C>aPggMg==-1</ChecksumCRC32C></CompleteMultipartUploadResult>
    127.0.0.1:9000

    • mc admin trace -call=ftp --verbose myminio

    127.0.0.1:9000 [FTP Stat] [2025-02-12T02:24:05.246] /device/6d60e92a-3e54-4ced-879c-e93423619d32 cmd=Stat param=/device/6d60e92a-3e54-4ced-879c-e93423619d32 source=[sftp-server-driver.go:397:(*sftpDriver).Filelist()] user=6d60e92a-3e54-4ced-879c-e93423619d32 1.477451ms
    127.0.0.1:9000 [FTP Put] [2025-02-12T02:24:05.256] /device/6d60e92a-3e54-4ced-879c-e93423619d32/GBR_NorthEastLive_GHCBSA1-01_20240909T121108+0000_15047.jpg user=6d60e92a-3e54-4ced-879c-e93423619d32 cmd=Put param=/device/6d60e92a-3e54-4ced-879c-e93423619d32/GBR_NorthEastLive_GHCBSA1-01_20240909T121108+0000_15047.jpg source=[sftp-server-driver.go:248:(*sftpDriver).Filewrite()] 78.230517ms 62 KiB

    • uploader.py code

    class SFTPClient(Collector):
    def __init__(self, response, args, **kwargs):
    super().__init__()

    self.conn = None
    self.response = dict(response)
    self.args = args

    keylist = ['hostname', 'port', 'username', 'password',
    'path', 'private_key_path', 'passphrase',
    'bucket_name', 'device_uuid', 'datetime']

    for key in keylist:
    setattr(self, key, self.response.get(key, None))

    self.upload = self.args.upload
    self.mode = self.args.mode
    self.interval = self.args.interval
    self.cam_id = self.args.cam_id

    if self.args.private_key_path is not None:
    logging.info(f"Update `private_key_path`: {self.private_key_path} -> {self.args.private_key_path}")
    self.private_key_path = self.args.private_key_path

    if self.args.passphrase is not None:
    logging.info(f"Update `passphrase`: {self.passphrase if self.passphrase is not None else None} -> {'*' * len(self.args.passphrase)}")
    self.passphrase = self.args.passphrase

    logging.info(f"Arguments: {self.args}")

    for key, val in kwargs.items():
    if key not in keylist:
    logging.error(f"Invalid key: `{key}`")
    return
    setattr(self, key, val)

    if self.mode == 'local':
    logging.info(f"Upload path: {self.upload}")
    elif self.mode == 'camera':
    logging.info(f"Camera id: {self.cam_id}(interval: {self.interval}")

    async def _create_client(self):
    try:
    '''
    if self.passphrase:
    private_key = asyncssh.read_private_key(self.private_key_path, password=self.passphrase)
    else:
    private_key = asyncssh.read_private_key(self.private_key_path)
    '''

    self.conn = await asyncssh.connect(
    host=self.hostname,
    port=int(self.port),
    username=self.username,
    client_keys=[self.private_key_path],
    passphrase=self.passphrase,
    known_hosts=None
    )

    logging.info(f"SSH connection established to {self.hostname}:{self.port} as {self.username}.")
    logging.info(f"connection: {self.conn} .")

    except (OSError, asyncssh.Error) as exc:
    logging.error(f"SSH/SFTP connection failed: {exc}")
    traceback.print_exc()

    return self.conn

    async def upload_from_local(self):
    is_file = True
    if not self.conn:
    self.conn = await self._create_client()

    if not self.conn:
    logging.error("Failed to create SSH connection. Aborting upload.")
    return

    try:
    async with self.conn.start_sftp_client() as sftp:
    logging.info(f"SFTP session opened: {sftp} .")

    if not os.path.isfile(self.upload): # is not file
    if os.path.isdir(self.upload): # is directory
    object_list = glob(os.path.join(self.upload, f"*"), recursive=True)
    self.upload = [x for x in object_list if os.path.isfile(x)]
    else: # is not file and not directory
    raise NotImplementedError
    else: # is file
    func = sftp.put

    logging.info(f"Uploading '{self.upload}' to '{self.path}'...")

    if isinstance(self.upload, list):
    for targetf in self.upload:
    if targetf is None:
    continue
    else:
    await sftp.put(targetf, self.path, block_size=32768)
    # await asyncio.sleep(self.interval)
    elif os.path.isfile(self.upload):
    await sftp.put(self.upload, self.path, block_size=32768)

    # await func(self.upload, self.path, block_size=4096)
    result = {"code": 200, "message": "success", "data": {"src": self.upload, "dst": self.path}}
    logging.info(json.dumps(result))

    except (OSError, asyncssh.Error, AttributeError) as failed_exc:
    logging.error(f"SSH/SFTP connection failed: {failed_exc}")
    traceback.print_exc()
    except asyncssh.sftp.SFTPNoSuchFile as notarget_exc:
    logging.error(f"SSH/SFTP cannot found file/path to upload: {notarget_exc}")
    traceback.print_exc()
    except asyncio.CancelledError as cancelled_exc:
    logging.error(f"Upload task cancelled: {cancelled_exc}")
    traceback.print_exc()
    except TimeoutError as timeout_exc:
    logging.error(f"Timeout: {timeout_exc}")
    traceback.print_exc()
    except Exception as e:
    logging.error(f"An unexpected error occurred: {e}")
    traceback.print_exc()

    finally:
    if self.conn:
    self.conn.close()
    await self.conn.wait_closed()
    self.conn = None
    logging.info("SSH connection closed.")


    This is my first time working on SFTP and it is not easy. Please help me. What else should I check?

    Continue reading...
    Stack, Fevereiro 12, 2025
    #1

Compartilhe esta Página