| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
We weren't doing the right thing when processing paginated data.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Don't crash on OSError when retrieving external subscriber info (e.g. if
there is a DNS lookup error).
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Trying to catch an elusive race condition, so add process-id to logging
information.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When we cannot match a commit to any patches, try to regenerate it again
with the histogram algorithm in case the original patch was sent using
that option.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
These headers carry important end-to-end signatures, so do not strip
them when processing archives.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When people send patches, their output will be dependent on their local
diff.algorithm setting. If they set it to something other than the
default ("myers"), we may fail to generate a matching patch from the
commit.
When we can't find a match, try to regenerate the patch using the
histogram algorithm and attempt the match again.
Reported-by: Jakub Kicinski <kuba@kernel.org>
Link: https://msgid.link/20230615100021.43d2d041@kernel.org
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The sync script can now use a git repository for external subscriber
data.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Added some features to retrieving messages from marc.info:
- earlier messages don't seem to have a message-id, so we fake it now
instead of completely ignoring these (they still serve a purpose)
- the "backoff" time now considers how much time has elapsed since the
last request to marc.info, so if we already spent a couple of seconds
checking a message for spam, we don't additionally sleep an extra second
- if the target mailbox already has messages in it, we will not
re-download those
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We're now grabbing a checksums file.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Small helper script to synchronize mlmmj lists with external subscriber
information.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Valid messages may come in with other actions than just
"sent_message_accepted" (e.g. "sent_message_moderated" or
"sent_message_not_subscriber"), so accept everything that contains the
message payload.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The message-id's in msgmap.sqlite3 are without <>, which we actually use
for mapping.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The things lists do to their messages is annoying.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Add a few small improvements to the list-archive-collector and
list-archive-maker lists, allowing to more easier deal with partial
archives.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Instead of duplicating code, just use a single instance of _save_emerg.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Should be largely backwards compatible with what we had before, but
allowing us to also troubleshoot occasional 500 errors.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We're seeing some 500 errors, and to debug them we need proper logging
and emergency message dumping.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This may change in the future, but for now treat .19 as the last minor
version before majver is incremented.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Add ability to specify list-ids as a globbling match (even if it's just
'*' for "everything").
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
It is periodically useful to save the archives as maildir instead of
mailbox. Add support for -asmaildir.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Add a note in the vcal output that the final release may actually end up
being another -rc instead.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We shouldn't crash out if we got a corrupted download. This script is
looking specifically at bad signatures -- a download that won't even
properly uncompress is probably not trying to sneak in badness.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We're migrating the nodes to their new homes on Equinix gen3 systems, so
adjust the list of edge sites accordingly.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
It worked, but it was a bug waiting to happen.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
- removes dependency on python-packaging
- switches to using logger calls instead of print()
- adds ability to generate .ics files
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Help people estimate kernel release dates.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When I switched to "git show" I didn't fully test whether it would
generate the same patchwork hashes as "git diff" proper. The few sample
tries I ran happened to do the right thing, but many other commits would
generate different hashes. Make sure we ignore anything that comes
before we find "diff " on the line.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Patchwork API 1.2 allows us to match by msgid, so update patchwork-bot
to use Link: trailers for additional matching when we are unable to
match by patchwork hash.
Additionally, reworks caching that wasn't actually saving us any time
and drops the use of git patch-id that we used in the absence of msgid
matching.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This reverts commit 8998144862b16cfa7a2ec07757d43ca269d59373.
Unfotunately, this change broke matching for other patches, so roll it
back to the way it was.
Reported-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We want to be able to share our configuration with others, which
requires that we keep secrets in a different file than the rest of the
logic.
Suggested-by: Kees Cook <kees@outflux.net>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
It helps to catch easy bugs when writing in PyCharm and other IDEs.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We used to need xmlrpc because API < 1.2 didn't expose ability to search
patches by hash. Now that it's available in API 1.2, we can get rid of
all xmlrpc calls and just leave the REST api support.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Finding a 2-character whitespace difference between patches with different
patchwork hashes strained my sanity, so I want to keep the tooling I
built to track it down. Using "--pwhash PATCH-ID", patchwork-bot will
compare the given patch in patchwork (and its hash) against the diff
from stdin (and its hash).
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20211202233836.2024510-3-keescook@chromium.org
|
|
It seems that there is some mismatch in diff output (or storage?) between
git and patchwork under conditions I haven't figured out (MUA or MTA
whitespace stripping?): patchwork appears to ignore empty context lines
(i.e. a line that is only a single space). As a result, the patchwork
hash will not match, and commits will not be identified in patchwork.
As an example of a commit in upstream that patchwork-bot cannot locate
in patchwork due to a different hash, due to the differing empty context
lines:
$ MSGID=20210911102818.3804-1-len.baker@gmx.com
$ wget -qO- https://patchwork.kernel.org/project/linux-hardening/patch/$MSGID/raw/ | \
grep '^ $' | wc -l
0
$ SHA=f11ee2ad25b22c2ee587045dd6999434375532f7
$ git diff ${SHA}~..${SHA} | \
grep '^ $' | wc -l
2
Teaching patchwork-bot to ignore empty context lines allows pwhashes to
match in these cases, which lets those patches get located again.
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20211202233836.2024510-2-keescook@chromium.org
|
|
To test patchwork-bot, it is helpful to create the initial database
without a certain number of the latest commits, so that on the next
execution, the "missing" commits will appear new, and will be used for
processing state changes. Add the --ancestors argument (defaulting to
0, the existing behavior), to provide this ability.
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20211201001126.4106635-6-keescook@chromium.org
|
|
Locate the real git location when operating on a worktree, allowing
multiple pw.db files to be stored in a single top-level git repo.
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20211201001126.4106635-5-keescook@chromium.org
|
|
To sanely use a single git repository with multiple worktrees, it must
be possible to limit the commits by branch (otherwise all commits are
visible for all worktrees). Add an optional "branch" argument for this
limit.
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20211201001126.4106635-4-keescook@chromium.org
|
|
Create the CACHEDIR if it is missing (e.g. the first time
git-patchwork-bot is executed).
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20211201001126.4106635-3-keescook@chromium.org
|
|
Expand the --help text slightly to remind users what things the
"housekeeping" option does.
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20211201001126.4106635-2-keescook@chromium.org
|
|
The refname is always going to be a branch, so remove "refs/heads/" in
all notices being sent.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
A few fixes in the patchwork-bot after intial trial runs. Change the
default template to mention who the committer is when sending submitter
notifications.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When running operations that require the shared cache database, make
sure that only a single instance of the bot is running.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
An oft-requested feature is showing who it was that committed the patch
to the repository. Add committer info to the summaries and submitter
notifications.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Set mode to 0755 for the bugzilla-junker script.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When recognizing merge commits, handle merge messages that do not
mention the refname of the remote.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The latest API offers some improvements we can use in the near future.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This is a helper script I use to keep tabs on spam posted to
bugzilla.kernel.org.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This is the backend that receives webhook events from groups.io and
feeds them into the local postfix queue for indexing with public-inbox.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Hyperkitty mangles things in similar, but slightly different ways from
pipermail.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
I always need to look up how to use the script, so have the script
provide a help command to show the valid commands.
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We always want all devices, regardless of how many pages we get back.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Show how many messages in each processed archive.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This creates refs/meta/origins entries for public-inbox archives.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When we iterate directly through messages, unicode or other parsing
errors cause us to BT on the entire mailbox. Iterating by key allows us
to only skip individual unparseable messages.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Without this change list-archive-maker just dies with an Exception
Signed-off-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20210414141235.26630-2-u.kleine-koenig@pengutronix.de
|
|
This is more pythonic and should also be quicker to execute.
Note there is an (intended) side effect of this change. If cc already
contains "twentyone@example.com" and pair[1] is "one@example.com", the
latter was detected as already contained in cc.
Signed-off-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20210414141235.26630-1-u.kleine-koenig@pengutronix.de
|
|
Add DCO from developercertificate.org and indicate that all
contributions must carry a Signed-off-by: line.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Both attest-patches and get-lore-mbox are now part of b4.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Tell people to send patches to tools@linux.kernel.org.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Minor bug with using an undefined value.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Indicate which keyring we are using when verifying signatures.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Mention that it can be run as a simple background process.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Make sure the admonition about not relying too much on sig-prover is
plainly visible.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Sig-prover is a tool I use to random-check mirrors for signs of tarball
corruption (or worse). It will download random kernel/git/etc tarballs
fron the frontends and verify them against the signatures, alerting when
there is a verification failure.
This script is not a guaranteed mechanism to detect intrusion -- an
attacker can defeat it by analyzing access patterns/IPs and serving
different content when it suspects that someone is running an automated
signature verification check. The script can probably be improved by
adding random delays between retrieving the tarball and the detached
signature, setting a referrer value, etc. However, even with added
measures, it will always act fairly predictably, so there will always
remain a way to defeat it.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Make it more generic without a pre-check. May not even be slower.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This is used by Greg KH to bring up a builder in Equinix Metal (and to
have it auto-ripped if he forgets to destroy it manually).
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Some users are running against stale cache problems with the checksums
file, so grab it directly from the origin servers instead.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Bundle files are now too big to be handled by cdn.kernel.org (Fastly),
so use canonical URLs that will redirect to mirrors.edge.kernel.org.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
I'm not sure why Python doesn't just return one or the other, but make
sure it's always a string before we try to operate on it.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Those do exist!
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Don't redefine fromstate, just take what is defined at the project
level.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
There can be changes to multiple refs that have identical ranges. Don't
attach multiple identical files in that case.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This is the hook that we are rolling out to generate the activity feed
suitable for historical (and forensic, should it ever be necessary)
analysis.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Move all operations for collecting remote archives into
list-archive-collector.py, adding a way to import marc.info archives. We
also add several other useful options to list-archive-collector:
- With -k, we can check if we already have a copy of that message on
lore.kernel.org and backfill many of the important headers that are
missing from pipermail and (especially) marc.info sources.
- With -s and prsence of /usr/bin/spamc, we'll check the message for
spam before accepting it
- The generated .mbx file can then be used with list-archive-maker.py to
create the final archive sources for importing into public-inbox.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We stopped auto-archiving as soon as we came across a page containing
patches past the cutoff date. However, for projects with a handful of
patches this meant that the entire page needed to fill up before we
auto-archived any eligible patches. This change goes through all patches
on the page before stopping.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Specifying "ccall" on a configuration will cc everyone who got mentioned
in the "to" or "cc" fields.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We don't need reposdir when we're running housekeeping tasks, so don't
require that argument.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Finally, provide an example config file that everyone's been wanting.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Lots of changes in this one:
- Drop support for python < 3.6
- Migrate to using a yaml file for configuration
- Implement some suggested features
Still testing and example config file to come.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Migrate to python3 and stop using git repos for config (too messy).
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Made some minor changes to make the script POSIX compliant
and some nitpicks to simplify the script.
Removed the if statements and replaced them with logical operators.
Also used default variable assignments to skip some checks to make
the script smaller.
Changed the temporary name of the bundle as some implementations of
mktemp (such as busybox) do not allow any name after the 'XXXXXXX'
part.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Setting references to multiple thread parents is discouraged by the RFCs
and causes unrelated messages to be grouped together by public-inbox.
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Because we go backwards from the --- line, multiple identical trailers
like "Signed-off-by" ended up reversed from the initial order. This
commit reverses them back.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Will now also accept trailers if:
- there's a full match for local part and partial match for domain part
of the email in from/trailer
- we find a comma in the name and match the trailer after rearranging
it
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We reject trailers that don't match the email address in the From:
field, but this can be too strict for our purposes. Relax the rule to
also allow matching by name.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Public-Inbox will return loosely matched threads, which is almost never
what we want for get-lore-mbox. This version implements strict thread
matching to make sure that we only pay attention to messages that have
clear references to the message-id passed as parameter.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Single patches are often posted in the same thread without any version
info, so auto-number revisions if they are newer and from the same
author (plus a slew of other sanity checks).
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
For hairier threads, we were wrongly guessing the series-starting patch
if one of the older non-numbered, non-versioned emails happened to
contain a diff (side-effect of trying to match more corner cases). We
will now properly replace the incorrect guess when coming across a
better match.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If there are problems with the thread that result in missing patches
(e.g. an omitted "v2" in the middle of the series), you can add a few
manual fixes to the mbox file before telling get-lore-mbox to make an
am-ready version out of it.
This also lets you bypass using lore.kernel.org completely if you want
to save patches out of your local mail client instead of always grabbing
them from lore.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Eventually, we won't be using the atom feed for this, but for now we
might as well not backtrace when we cannot parse it.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
- Attempt to deliver via mail.kernel.org:587
- Accept patch from Kees Cook to set custom From: header
- If we find a cover letter, set attestation subject to match
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Unless we're running with -F that ignores From/UID mismatches, we will
now use the matching UID in the Attestation-by trailer.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We need to see who is the person attesting the patches.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This is handy for use with attest-patches, though eventually attestation
will be rolled into get-lore-mbox itself.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
- Skip messages not containing patches (like cover letters)
- Add option to not fast-exit (-X), which is handy for testing
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
- Add an option to ignore From/UID mismatches
- Add an option to force TOFU/default-good trust model
- Add an option to load attestation data from a local file instead of
always querying lore.kernel.org
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Tools like git-send-email don't preserve the Date: header as in the body
generated by git-format-patch, so we cannot rely on it for attestation
purposes. See:
https://lore.kernel.org/git/20191213015753.GA14249@generichostname/T/#u
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This is a proof-of-concept script for submitting patch attestation. It
should not be used without more work, as it almost certainly doesn't
consider a bunch of potentially malicious corner cases that would give
wrong attestation results.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When there is a quote in the subject, we need to remove it in order for
the xapian "phrase-in-quotes" search to work.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
- Deal with exchange not setting in-reply-to and only setting References
- Don't BT when dealing with follow-ups to missing cover letters
- Be more forgiving for patches missing [PATCH] in the subject
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Properly handle cases when the cover letter is missing from the thread.
Reported-by: Rob Herring <robh@kernel.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Handle special case when patches don't contain ---/+++ because they are
a "GIT binary patch".
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When we need to do multiline regexes, make sure we don't match
linebreaks when we mean for a regex to only match a single line.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When doing trailer collating, don't break if there are missing messages
in the thread.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If we find a text/plain part, but also anything matching text/plain or
*/x-patch that contains a diff, then we'll use the patch containing a
diff instead.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Now should properly handle commas, e.g.: [PATCH 01/02, FOO].
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Getting a message as_bytes() before adding it to the am-ready mailbox
allows us to pass the policy that avoids triggering some of the odder
bugs in older versions of python.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This is mostly to avoid bogus matches, but adds a small layer of sanity
checking.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Also, makes trailer-order configurable via ~/.gitconfig, e.g.:
[get-lore-mbox]
# remember to end with ,*
trailer-order=link*,fixes*,cc*,reported*,suggested*,original*,co-*,tested*,reviewed*,acked*,signed-off*,*
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Instead of using requests directly, open a session instead. This makes
it both faster and allows us to set global user-agent header.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
By popular demand, add -Q, which will write out a directory full of
patches and the "series" file that can be manipulated using quilt.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Switched away from using git-interpret-trailers in order to offer a way
to reorder trailers and to stick additional trailers within the same
block of others already present (if any).
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Specifically, be forgiving of someone finger-fumbling [PATCH v2 xx/nn]
as [PATCHv2 xx/nn].
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
As the feature set grew, it became obvious that the structure needed to
be less hacky (initial code was just barely beyond a proof of concept).
This moves most of the am-mangling code into classes where it makes much
more sense, plus makes debugging easier.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Actually pay attention to what the charset says in the message headers
for the few holdouts who are still sending things as iso-8859-1.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We're already returning Nones when something isn't doing the right
thing, but we aren't properly checking for them. This fixes two of the
reported cases.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Work around for a logic bug when trying to find non-standard cover
letters.
Reported-by: Greg Kroah-Hartman <greg@kroah.com>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Found an In-Reply-To that was not just a <foo>, so handle those cases
better with a regex.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
You can now pass "-c/--check-newer-revisions" to query lore.kernel.org
for never revisions to a series.
I'm not overly fond of how this is done -- it will need rewriting once
lore/public-inbox supports proper search-based API. However, it does its
job and is what maintainers are asking for.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Make sure that all trailers are unique by making a set out of them.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When a trailer is sent in reply to the cover letter, give an option to
apply it to all patches. We don't do it by default because sometimes
developers include verbal caveats like:
For patches 1-5, you can add:
Reviewed-by: Foo Foofski <foo@example.com>
In this case blindly applying this trailer to all patches would convey
wrong information.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Some of the more complex mime messages were returning wrong parts
instead of patch content, so make sure that when we're looking for a
patch, we get the patch.
Also, call git interpret-trailer once for multiple trailers added.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We aren't sending these via smtp, so remove all legacy encoding schemes
from the headers (base64, quoted-printable, etc). We always write 8bit
utf-8.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Notable changes:
- Saves the cover letter as a separate file, to make it "git am"
operation more streamlined (no need to --skip right away when a cover
letter is included)
- More reliable handling for base-commit information
- Fixes single-patch submissions
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
- Give thread summary after saving (without -a)
- Be less verbose by default when saving with -a
- Remove raw inbox after making an am-version
- Remove most of the header bloat in the am-ready mbox
- Remove any mime data from the am-ready mbox and save as 8bit
(this makes manual editing much simpler)
- Try to figure out what the cover letter is, even if it's not using the
00/NN notation
- Try a few more ways of figuring out base-commit information
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Decode payloads from base64 before collating trailers.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This gets you full threads from lore.kernel.org using just a single
message-id of a message anywhere in that thread. It can also create mbox
files ready for use with "git am", with all trailers tallied up.
Needs more testing, but generally works fairly well.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This is an extremely simple implementation of a sendmail wrapper to
create public-inbox developer feeds. It only needs bash and git and is
written to be used with git-send-email.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If we prepped new shard, always commit the db transaction in case there
are no new changes to the previous epoch between the runs.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
It would be best if we could just search by series that contain a
pull_url field, but we can't. So, relax this search and iterate through
a larger set, but catch more flies.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
- don't crash when pagedata is None (no more results)
- fix regex used to recognize pull request merges
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This allows us to send notification cc's to the list without needing to
specify the list email in the config file.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This is not as good as pr-trackert-bot, but should cover most common
cases. Patchwork needs to recognize some of the less common formats for
pull requests (the way we do with pr-tracker-bot), and when it does this
should automatically improve as well.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The following message did not get proper cc's set in the body of the
message because Cc: is spread across multiple headers. This should fix
the problem.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Minisign is still a bit of a second fiddle to PGP, but at least this
puts it on the map. To enable, at least the following bit needs to be
set either in ~/.gitconfig or in .git/config of the repository where the
signing is done:
[archive-signer]
use-minisign = yes
Other options you can add to that are:
minisign-key = path/to/minisign.key
minisign-gpg-passphrase = path/to/minisign-passphrase.gpg
The latter option is handy if you don't want to remember the minisign
passphrase and would instead prefer gpg-agent to decrypt it and provide
to minisign when needed.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Sometimes a new pull request will come in as a response to an existing
pull request that got turned down. Check messages matching those
subjects as well.
See: https://lore.kernel.org/lkml/CAJ-EccOqmmrf2KPb7Z7NU6bF_4W1XUawLLy=pLekCyFKqusjKQ@mail.gmail.com/T
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This reduces log bloat significantly.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If we only have [\w+], then we don't match prefixes like [for-next,v2],
and therefore miss series we can auto-supersede. Do a lazy wildcard
match instead.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This does a much better job of handling shard roll-over.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We were not using proper order when looking up remote refs, for details
see this thread:
https://lore.kernel.org/lkml/CAHk-=wg9yvP3hGVRSnc=8qoj0g7_zt=_2FjmgSAu-0DViyFtaw@mail.gmail.com
This fix changes the order to properly match heads and be more correct
when handling shortnames in general.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The LKML is about to roll over from shard 6 to shard 7, so add some
logic to properly handle situations where we have tracked messages
referring to commit-id's from the previous shard. It's a bit hacky, but
it should do the job.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This ensures that we don't duplicated To/Cc addresses when processing
potentially multiple header entries.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
More mail will be matched this way, so that messages with mixed-case
list addresses (e.g. Linux-MM@kvack.org) are included.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
|
|
Emails should have at most one To: and one Cc: header, but sometimes
malformed ones have more than that, causing 'notourlist' false
positives and leaving legitimate messages out of the archive.
Collapse multiple identical headers into one.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
|
|
Sometimes a message is sent to a mailing list using the Resent-to:
header. Check for this so these types of messages aren't missed.
Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
|
|
A not entirely unexpected event happened when a Pull Request showed up
in a public-inbox repo after it's already been merged into the
repository. In this case, it was due to mailserver delay, but this will
likely happen again in the future simply due to the nature of email and
mailing lists.
On the "mail thank-yous" run, instead of iterating through the new
commits since the last run, go through the outstanding PRs and see if we
can find them in the repository by running "git branch --contains
$commit_id". If we get a refname, then that PR has been applied and we
can thank the person for it.
This is a bit heavier than the previous incarnation, since we are now
checking this on every run instead of only when we find new revs in any
of the heads. However, this is still sufficiently light not to worry
about it too much.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
I'm also testing the push hook using this repo.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This allows verifying that all commits are signed either from the last
annotated tag, or all the way to an arbitrary commit object in the past.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Needs initial test runs, but looks promising.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Usually, patchwork will strip those if it recognizes them as prefixes,
but it will miss some of them. Be more agressive than patchwork here.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Patchwork sneakily replaces spaces with dashes in state names for REST
API purposes.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This makes the linter happy and makes the shell scripts more uniform.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Reported-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We're now matching too many things, so rework a bit to better eliminate
false-positives.
Also, fix locking so it's actually doing what it needs to do.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Fixes:
https://lore.kernel.org/lkml/20190109180501.12290.85607.pr-tracker-bot@pdx-korg-gitolite-1.ci.codeaurora.org/
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If a patch is marked as "Mainlined", then we don't want to go back to
mark it as "Accepted". Unfortunately, patchwork does not expose the
"work required" field of the state, so we can't rely on that. However,
for the vast majority of cases, we only want to consider patches that
are either "new" or "under review" (and we can tweak this via the config
value if we need to add more states to that).
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
There are cases where devs aren't using v1, v2 quite as patchwork
expects them to do it, e.g. there will be:
[foo] Fix foo with bar
followed by:
[foo,v1] Fix foo with bar
These both would be marked as "v1" by patchwork, so use dates to figure
out which series supersedes which regardless of prefix markers.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
An email should theoretically only have a single cc header, but
sometimes that's no the case.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Fixes traceback for reportless housekeeping.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Some people don't want to litter their cc's with notify@kernel.org, so
let them set a special header instead.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
And, really, multipart-html requests, even though we won't see them on a
VGER list.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Not sure how this is triggered -- I'd expect the patch_id to always be
in that list. Pretty sure this is just a manifest of another bug.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We're now only using the xmlrpc API to query for hash matches --
everything else is done via the REST API.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The patches can arrive out of order, so patch_id may not represent the
order in the series.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
