add length to progress bar

[xsuchy]

so the progress bar shows real progress

Fixes #3342

Tasks

• Reviewed contribution guidelines
• PR is descriptively titled 📑 and links the original issue above 🔗
• Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
  Run tests locally to check for errors.
  The testsuite is broken for me and produce lots of not relevant errors. I will appreciate testing.
• Commits are in uniquely-named feature branch and has no merge conflicts 📁
• [-] Updated documentation pages (if applicable)
• Updated CHANGELOG.rst (if applicable)

[pombredanne]

Miroslav and thanks!
There is an issue wrt. to making this work with large codebase. Even a linux kernel tarball may be problematic... we may need to find a better way.

[pombredanne]

This will be a problem when we have a codebase with 10 million files, we may need to find a better way? ... otherwise the system will literally be stalled for minutes before displaying a progress bar.

[xsuchy]

For the kernel, the delay is 45 sec until it traverses 200k files. While the scan itself runs several hours.
With -q there is no delay.

[pombredanne]

Why the times 2?

[xsuchy]

Because #3344 Otherwise the progressbar go to 170/170 and then continue to 340/170. This *2 can be removed when we find when we find where is the count done twice. I have a suspicion, but it needs more testing.

[xsuchy]

Several things I considered as an option:

1. instead resources = list(((r.location, r.path) for r in codebase.walk() if r.is_file)) use simpler resources = list((1 for r in codebase.walk() if r.is_file)) because we just need the count. This is consume less memory, but does not give any measurable time.

2. pass the resources to scan_codebase so we do not need to count it twice. Hmm. I will do this. PR updated.

[xsuchy]

I tried to improve the performance of walk() in aboutcode-org/commoncode#78

[armijnhemel]

Looking at the code I don't see that this would fix #3342. I don't see that the number of files that will be processed is displayed.

[xsuchy]

Looking at the code I don't see that this would fix #3342. I don't see that the number of files that will be processed is displayed.

The line with resources = list(resources) is the number of files (times two for some reason).

The line with 'scans = progress_manager(scans)` set up progress meter. But because constructor does not have the number of items, it does not show the real progress.

The line scans.length = len(resources)*2 sets the the lentgs of the queue in the progress meter so it then shows the real progress and number of processed items.

[armijnhemel]

Looking at the code I don't see that this would fix #3342. I don't see that the number of files that will be processed is displayed.

The line with resources = list(resources) is the number of files (times two for some reason).

The line with 'scans = progress_manager(scans)` set up progress meter. But because constructor does not have the number of items, it does not show the real progress.

The line scans.length = len(resources)*2 sets the the lentgs of the queue in the progress meter so it then shows the real progress and number of processed items.

Yes but in #3342 I don't care about the progress meter, I just want to see the number of files that will be scanned :-)

[xsuchy]

This is how it looks like: You can see the number of files being scanned on the right side of progress bar as $DONE/$TOTAL.

And when I was doing this screenshot, I rebased this PR.

[armijnhemel]

This is how it looks like: You can see the number of files being scanned on the right side of progress bar as $DONE/$TOTAL.

And when I was doing this screenshot, I rebased this PR.

Oh, this would be great. @pombredanne, maybe some time should be spent finding out where the double counting happens? I tried to look for it but I couldn't find it.

[xsuchy]

Rebased.

[xsuchy]

Can you please consider a new round of review?
With aboutcode-org/commoncode#78 merged, the penalty at the startup is really minimal.

[xsuchy]

I removed the *2 because I found the cause. It is fixed in aboutcode-org/commoncode#100, so the length is now correct.
Please review.

[armijnhemel]

so the progress bar shows real progress

Fixes #3342

Actually it would not fix that issue, as I am asking for something else. What I am asking for in that issue is to display a total number of files that will be scanned, so I have an indication of how many files in total will be scanned. This PR fixes a different issue.