Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Register
  • Sign in
  • slapos slapos
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Merge requests 129
    • Merge requests 129
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Jobs
  • Commits
Collapse sidebar
  • nexedinexedi
  • slaposslapos
  • Merge requests
  • !1109

component/python-2.7: Lib/subprocess: Speedup close_fds=True

  • Review changes

  • Download
  • Patches
  • Plain diff
Closed Kirill Smelkov requested to merge kirr/slapos:y/py27-subproc-closefd-fast into master Dec 24, 2021
  • Overview 8
  • Commits 1
  • Pipelines 0
  • Changes 2

@romain reports that Popen(close_fds=True) is slow on py2. Let's semantically backport from py3 how to close only actually opened file descriptors instead of whole 3..ulimit -n range.

Attached test benchmark shows the following results with ulimit -n=65K:

Before this patch:

$ ./bin/python2.7 ~/x.py
close_fds=False:
0.001251 s/call
0.001337 s/call
0.001486 s/call

close_fds=True:
0.017973 s/call
0.018152 s/call
0.018204 s/call

After the patch:

$ ./bin/python2.7 ~/x.py
close_fds=False:
0.001391 s/call
0.001416 s/call
0.001570 s/call

close_fds=True:
0.001469 s/call
0.001479 s/call
0.001491 s/call

i.e. ~12x speedup.

References on this subject are in the patch itself.

The test benchmark is below:

---- 8< ----

import timeit
from subprocess import check_call

def f():
    check_call(['true'], close_fds=False)
def g():
    check_call(['true'], close_fds=True)

N=3
n=100
print 'close_fds=False:'
for i in range(N):
    print '%.6f s/call' % (timeit.timeit(f, number=n) / n)
print
print 'close_fds=True:'
for i in range(N):
    print '%.6f s/call' % (timeit.timeit(g, number=n) / n)

/helped-by @jm

Edited Dec 30, 2021 by Kirill Smelkov
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: y/py27-subproc-closefd-fast
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7