[Python] Azure Pipelines run into a arguments too long error

I am running an azure pipeline, where I execute some python scripts. Using a python script, I generate a txt file (A.txt) which contains a list of strings. For eg: The contents of A.txt is ["A", "B", "C"......]

I have a file now which creates a list of 40000 strings. I have published as a pipeline artifact. I execute a python script python3 execute.py --file A.txt , I get arguments tooo long error:

2025-06-13T09:15:51.0744906Z ##[debug]Agent environment resources - Disk: / Available 36787.11 MB out of 64201.73 MB, Memory: Used 964.00 MB out of 7952.00 MB, CPU: Usage NaN%
2025-06-13T09:15:52.8721205Z ##[debug]Using node path: /home/azureadmin/azureadminagent/externals/node20_1/bin/node
2025-06-13T09:15:52.8941611Z ##[error]An error occurred trying to start process '/home/azureadmin/azureadminagent/externals/node20_1/bin/node' with working directory '/home/azureadmin/azureadminagent/_work/28/s'.
Argument list too long
2025-06-13T09:15:52.8951056Z ##[debug]System.ComponentModel.Win32Exception (7): An error occurred trying to start process '/home/azureadmin/azureadminagent/externals/node20_1/bin/node' with working directory
'/home/azureadmin/azureadminagent/_work/28/s'. Argument list too long
at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd,
Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
at System.Diagnostics.Process.Start()
at Microsoft.VisualStudio.Services.Agent.Util.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment,
Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen,
Boolean highPriorityProcess, Boolean continueAfterCancelProcessTreeKillAttempt, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/src/Agent.Sdk/ProcessInvoker.cs:line 299
at Microsoft.VisualStudio.Services.Agent.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment,
Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen,
Boolean highPriorityProcess, Boolean continueAfterCancelProcessTreeKillAttempt, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/src/Microsoft.VisualStudio.Services.Agent/ProcessInvoker.cs:line 329
at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.DefaultStepHost.ExecuteAsync(String workingDirectory, String fileName, String arguments,
IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, Boolean continueAfterCancelProcessTreeKillAttempt,
TimeSpan sigintTimeout, TimeSpan sigtermTimeout, Boolean useGracefulShutdown, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/src/Agent.Worker/Handlers/StepHost.cs:line 85
at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.NodeHandler.RunAsync() in /mnt/vss/_work/1/s/src/Agent.Worker/Handlers/NodeHandler.cs:line 267
at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.RunAsyncInternal() in /mnt/vss/_work/1/s/src/Agent.Worker/TaskRunner.cs:line 447
at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.RunAsync() in /mnt/vss/_work/1/s/src/Agent.Worker/TaskRunner.cs:line 76
at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken) in /mnt/vss/_work/1/s/src/Agent.Worker/StepsRunner.cs:line 264
2025-06-13T09:15:52.8953421Z ##[section]Finishing: listing out files


I observe the above error even if I do a ls -la on the directory in the pipeline.


In another approach, I manually added this file A.txt with the same contents to the repository directly, and executed the python3 execute.py --file A.txt via the pipeline. That works fine.


When the A.txt has less than 20K entries, it works fine.

What is going wrong here? Suggestions are welcome

Continue reading...