openSUSE Build Service

Overview Repositories Revisions Requests Users Attributes Meta

File b4-0.6.2+4.obscpio of Package b4 (Revision 0b9d4f36093b893afcbad0c766bbc24d)

Currently displaying revision 0b9d4f36093b893afcbad0c766bbc24d , Show latest

07070100000000000081A40000273B00000064000000015FEA2051000046AC000000000000000000000000000000000000001300000000b4-0.6.2+4/COPYING                    GNU GENERAL PUBLIC LICENSE
                       Version 2, June 1991

Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
 Everyone is permitted to copy and distribute verbatim copies
 of this license document, but changing it is not allowed.

Preamble

The licenses for most software are designed to take away your
freedom to share and change it.  By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users.  This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it.  (Some other Free Software Foundation software is covered by
the GNU Lesser General Public License instead.)  You can apply it to
your programs, too.

When we speak of free software, we are referring to freedom, not
price.  Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.

To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.

For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have.  You must make sure that they, too, receive or can get the
source code.  And you must show them these terms so they know their
rights.

We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.

Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software.  If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.

Finally, any free program is threatened constantly by software
patents.  We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary.  To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.

The precise terms and conditions for copying, distribution and
modification follow.

GNU GENERAL PUBLIC LICENSE
   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License.  The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language.  (Hereinafter, translation is included without limitation in
the term "modification".)  Each licensee is addressed as "you".

Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope.  The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.

1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.

You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.

2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:

a) You must cause the modified files to carry prominent notices
    stating that you changed the files and the date of any change.

b) You must cause any work that you distribute or publish, that in
    whole or in part contains or is derived from the Program or any
    part thereof, to be licensed as a whole at no charge to all third
    parties under the terms of this License.

c) If the modified program normally reads commands interactively
    when run, you must cause it, when started running for such
    interactive use in the most ordinary way, to print or display an
    announcement including an appropriate copyright notice and a
    notice that there is no warranty (or else, saying that you provide
    a warranty) and that users may redistribute the program under
    these conditions, and telling the user how to view a copy of this
    License.  (Exception: if the Program itself is interactive but
    does not normally print such an announcement, your work based on
    the Program is not required to print an announcement.)

These requirements apply to the modified work as a whole.  If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works.  But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.

Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.

In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.

3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:

a) Accompany it with the complete corresponding machine-readable
    source code, which must be distributed under the terms of Sections
    1 and 2 above on a medium customarily used for software interchange; or,

b) Accompany it with a written offer, valid for at least three
    years, to give any third party, for a charge no more than your
    cost of physically performing source distribution, a complete
    machine-readable copy of the corresponding source code, to be
    distributed under the terms of Sections 1 and 2 above on a medium
    customarily used for software interchange; or,

c) Accompany it with the information you received as to the offer
    to distribute corresponding source code.  (This alternative is
    allowed only for noncommercial distribution and only if you
    received the program in object code or executable form with such
    an offer, in accord with Subsection b above.)

The source code for a work means the preferred form of the work for
making modifications to it.  For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable.  However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.

If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.

4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License.  Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.

5. You are not required to accept this License, since you have not
signed it.  However, nothing else grants you permission to modify or
distribute the Program or its derivative works.  These actions are
prohibited by law if you do not accept this License.  Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.

6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions.  You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.

7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License.  If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all.  For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.

If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.

It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices.  Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.

This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.

8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded.  In such case, this License incorporates
the limitation as if written in the body of this License.

9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time.  Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.

Each version is given a distinguishing version number.  If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation.  If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.

10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission.  For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this.  Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.

NO WARRANTY

11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.

12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.

END OF TERMS AND CONDITIONS

How to Apply These Terms to Your New Programs

If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.

To do so, attach the following notices to the program.  It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.

<one line to give the program's name and a brief idea of what it does.>
    Copyright (C) <year>  <name of author>

This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation; either version 2 of the License, or
    (at your option) any later version.

This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

You should have received a copy of the GNU General Public License along
    with this program; if not, write to the Free Software Foundation, Inc.,
    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

Also add information on how to contact you by electronic and paper mail.

If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:

Gnomovision version 69, Copyright (C) year name of author
    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
    This is free software, and you are welcome to redistribute it
    under certain conditions; type `show c' for details.

The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License.  Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.

You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary.  Here is a sample; alter the names:

Yoyodyne, Inc., hereby disclaims all copyright interest in the program
  `Gnomovision' (which makes passes at compilers) written by James Hacker.

<signature of Ty Coon>, 1 April 1989
  Ty Coon, President of Vice

This General Public License does not permit incorporating your program into
proprietary programs.  If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library.  If this is what you want to do, use the GNU Lesser General
Public License instead of this License.
07070100000001000081A40000273B00000064000000015FEA205100000034000000000000000000000000000000000000001700000000b4-0.6.2+4/MANIFEST.ininclude COPYING
include man/*.rst
include *.example
07070100000002000081A40000273B00000064000000015FEA2051000012B7000000000000000000000000000000000000001600000000b4-0.6.2+4/README.rstB4 tools
========
This is a helper utility to work with patches made available via a
public-inbox archive like lore.kernel.org. It is written to make it
easier to participate in a patch-based workflows, like those used in
the Linux kernel development.

The name "b4" was chosen for ease of typing and because B-4 was the
precursor to Lore and Data in the Star Trek universe.

See man/b4.5.rst for more information.

Installing
----------
To install from pypi::

python3 -m pip install --user b4

Upgrading
---------
If you previously installed from pypi::

python3 -m pip install --user --upgrade b4

Running from the checkout dir
-----------------------------
If you want to run from the checkout dir without installing the python
package, you can use the included ``b4.sh`` wrapper. You can set it as
an alias in your .bash_profile::

alias b4="$HOME/path/to/b4/b4.sh"

Setting up a symlink should also be possible.

Patch attestation (EXPERIMENTAL)
--------------------------------
Starting with version 0.6, b4 implements in-header patch attestation,
following the approach proposed here:

https://git.kernel.org/pub/scm/linux/kernel/git/mricon/patch-attestation-poc.git/tree/README.rst

At this time, only PGP mode is implemented, but further work is expected
in future versions of b4.

Attesting your own patches
~~~~~~~~~~~~~~~~~~~~~~~~~~
Patch attestation is done via message headers and stays out of the way
of usual code submission and review workflow. At this time, only
maintainers using b4 to retrieve patches and patch series will benefit
from patch attestation, but everyone is encouraged to submit
cryptographic patch attestation with their work anyway, in hopes that it
becomes a common and widely used procedure.

To start attesting your own patches:

1. Make sure you have b4 version 0.6.0 or above:
   ``b4 --version``
2. If you don't already have a PGP key, you can follow the following
   guide on how to generate it:
   https://www.kernel.org/doc/html/latest/process/maintainer-pgp-guide.html
3. It is strongly recommended to use ed25519 as your signing key
   algorithm, as it will result in much smaller signatures, preventing
   unnecessary email header bloat.
4. Make sure your ``user.email`` and ``user.signingkey`` are set either
   globally, or in the repository you will be using for attestation.
5. Add the ``sendemail-validate`` hook to each repository you want
   enabled for attestation, with the following single line of content as
   the hook body:
   ``b4 attest $1``.

If you are using b4 from git checkout, you can use a symlink instead::

ln -s path/to/b4/hooks/sendemail-validate-attestation-hook \
        .git/hooks/sendemail-validate

(Note, that there's a second "E" in send*E*mail.)

Next time you run ``git send-email``, b4 will automatically add
attestation headers to all patches before they go out.

Verifying attestation on received patches
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are three attestation verification policies in b4:

- check (default)
- softfail
- hardfail

The default "check" policy is look for any available attestation and try
to verify it. If verification fails, b4 will not output any errors, but
will not show verification checkmarks either.

In "softfail" mode, any verification errors will be prominently
displayed, but b4 will still generate the .mbx file with patches.

The "hardfail" mode will show verification errors and exit without
generating the .mbox file with patches.

You can set the preferred policy via the git configuration file::

[b4]
      attestation-policy = softfail

Using with mutt
~~~~~~~~~~~~~~~
You can show patch attestation data with mutt, using the following
configuration parameters::

set display_filter="b4 -q attest -m"
    ignore *
    unignore from date subject to cc list-id:
    unignore x-patch-hashes: x-patch-sig:
    unignore attested-by: attestation-failed:

When displaying a message containing in-header PGP attestation
signatures, mutt will display either the "Attested-By" or the
"Attestation-Failed" headers, e.g.::

Date: Mon, 23 Nov 2020 13:38:50 -0500
    From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
    To: mricon@kernel.org
    Subject: [PATCH 3/5] Fix in-header attestation code
    Attested-By: Konstantin Ryabitsev <konstantin@linuxfoundation.org> (pgp: B6C41CE35664996C)

or::

Date: Mon, 23 Nov 2020 13:38:48 -0500
    From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
    To: mricon@kernel.org
    Subject: [PATCH 1/5] Add not very simple dkim key caching
    Attestation-Failed: signature failed (commit message, patch metadata)

Support
-------
For support or with any other questions, please email
tools@linux.kernel.org, or browse the list archive at
https://linux.kernel.org/g/tools.
07070100000003000041ED0000273B00000064000000025FEA205100000000000000000000000000000000000000000000000E00000000b4-0.6.2+4/b407070100000004000081ED0000273B00000064000000015FEA2051000000EC000000000000000000000000000000000000001100000000b4-0.6.2+4/b4.sh#!/usr/bin/env bash
#
# Run b4 from a git checkout.
#

REAL_SCRIPT=$(realpath -e ${BASH_SOURCE[0]})
SCRIPT_TOP="${SCRIPT_TOP:-$(dirname ${REAL_SCRIPT})}"

exec env PYTHONPATH="${SCRIPT_TOP}" python3 "${SCRIPT_TOP}/b4/command.py" "${@}"
07070100000005000081A40000273B00000064000000015FEA20510001736B000000000000000000000000000000000000001A00000000b4-0.6.2+4/b4/__init__.py# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright (C) 2020 by the Linux Foundation
import subprocess
import logging
import hashlib
import re
import sys
import gzip
import os
import fnmatch
import email.utils
import email.policy
import email.header
import requests
import urllib.parse
import datetime
import time
import shutil
import mailbox
import pwd
import base64

from pathlib import Path
from tempfile import mkstemp, TemporaryDirectory
from contextlib import contextmanager

from email import charset
charset.add_charset('utf-8', None)
emlpolicy = email.policy.EmailPolicy(utf8=True, cte_type='8bit', max_line_length=None)

try:
    import dns.resolver
    import dkim

can_dkim_verify = True
    _resolver = dns.resolver.get_default_resolver()
except ModuleNotFoundError:
    can_dkim_verify = False
    _resolver = None

__VERSION__ = '0.6.3-dev'

logger = logging.getLogger('b4')

HUNK_RE = re.compile(r'^@@ -\d+(?:,(\d+))? \+\d+(?:,(\d+))? @@')
FILENAME_RE = re.compile(r'^(---|\+\+\+) (\S+)')

PASS_SIMPLE = '[P]'
WEAK_SIMPLE = '[D]'
FAIL_SIMPLE = '[F]'
PASS_FANCY = '\033[32m\u2714\033[0m'
WEAK_FANCY = '\033[32m\u2713\033[0m'
FAIL_FANCY = '\033[31m\u2717\033[0m'

HDR_PATCH_HASHES = 'X-Patch-Hashes'
HDR_PATCH_SIG = 'X-Patch-Sig'

# You can use bash-style globbing here
WANTHDRS = [
    'sender',
    'from',
    'to',
    'cc',
    'subject',
    'date',
    'message-id',
    'resent-message-id',
    'reply-to',
    'in-reply-to',
    'references',
    'list-id',
    'errors-to',
    'x-mailing-list',
    'resent-to',
]

# You can use bash-style globbing here
# end with '*' to include any other trailers
# You can change the default in your ~/.gitconfig, e.g.:
# [b4]
#   # remember to end with ,*
#   trailer-order=link*,fixes*,cc*,reported*,suggested*,original*,co-*,tested*,reviewed*,acked*,signed-off*,*
#   (another common)
#   trailer-order=fixes*,reported*,suggested*,original*,co-*,signed-off*,tested*,reviewed*,acked*,cc*,link*,*
#
# Or use _preserve_ (alias to *) to keep the order unchanged

DEFAULT_TRAILER_ORDER = '*'

LOREADDR = 'https://lore.kernel.org'

DEFAULT_CONFIG = {
    'midmask': LOREADDR + '/r/%s',
    'linkmask': LOREADDR + '/r/%s',
    'trailer-order': DEFAULT_TRAILER_ORDER,
    # off: do not bother checking attestation
    # check: print an attaboy when attestation is found
    # softfail: print a warning when no attestation found
    # hardfail: exit with an error when no attestation found
    'attestation-policy': 'check',
    # "gpg" (whatever gpg is configured to do) or "tofu" to force tofu mode
    'attestation-trust-model': 'gpg',
    # strict: must match one of the uids on the key to pass
    # loose: any valid and trusted key will be accepted
    'attestation-uid-match': 'loose',
    # How many days before we consider attestation too old?
    'attestation-staleness-days': '30',
    # Should we check DKIM signatures if we don't find any other attestation?
    'attestation-check-dkim': 'yes',
    # We'll use the default gnupg homedir, unless you set it here
    'attestation-gnupghome': None,
    # Do you like simple or fancy checkmarks?
    'attestation-checkmarks': 'fancy',
    # How long to keep things in cache before expiring (minutes)?
    'cache-expire': '10',
    # Used when creating summaries for b4 ty
    'thanks-commit-url-mask': None,
    # See thanks-pr-template.example
    'thanks-pr-template': None,
    # See thanks-am-template.example
    'thanks-am-template': None,
    # If this is not set, we'll use what we find in 
    # git-config for gpg.program, and if that's not set,
    # we'll use "gpg" and hope for the better
    'gpgbin': None,
}

# This is where we store actual config
MAIN_CONFIG = None
# This is git-config user.*
USER_CONFIG = None

# Used for tracking attestations we have already looked up
ATTESTATIONS = list()
# Used for keeping a cache of subkey lookups to minimize shelling out to gpg
SUBKEY_DATA = dict()
# Used for storing our requests session
REQSESSION = None
# Indicates that we've cleaned cache already
_CACHE_CLEANED = False
# Used for dkim key lookups
_DKIM_DNS_CACHE = dict()

class LoreMailbox:
    def __init__(self):
        self.msgid_map = dict()
        self.series = dict()
        self.covers = dict()
        self.trailer_map = dict()
        self.followups = list()
        self.unknowns = list()

def __repr__(self):
        out = list()
        for key, lser in self.series.items():
            out.append(str(lser))
        out.append('--- Followups ---')
        for lmsg in self.followups:
            out.append('  %s' % lmsg.full_subject)
        out.append('--- Unknowns ---')
        for lmsg in self.unknowns:
            out.append('  %s' % lmsg.full_subject)

return '\n'.join(out)

def get_by_msgid(self, msgid):
        if msgid in self.msgid_map:
            return self.msgid_map[msgid]
        return None

def backfill(self, revision):
        if revision in self.covers and self.covers[revision] is not None:
            patch = self.covers[revision]
        else:
            # Find first non-None member in patches
            lser = self.series[revision]
            patch = None
            for patch in lser.patches:
                if patch is not None:
                    break
        logger.info('---')
        logger.info('Thread incomplete, attempting to backfill')
        cachedir = get_cache_dir()
        listmap = os.path.join(cachedir, 'lists.map.lookup')
        if not os.path.exists(listmap):
            # lists.map is a custom service running on lore.kernel.org, so it is
            # meaningless to make this a configurable URL
            session = get_requests_session()
            resp = session.get('https://lore.kernel.org/lists.map')
            if resp.status_code != 200:
                logger.debug('Unable to retrieve lore.kernel.org/lists.map')
                return
            content = resp.content.decode('utf-8')
            with open(listmap, 'w') as fh:
                fh.write(content)
        else:
            with open(listmap, 'r') as fh:
                content = fh.read()

projmap = dict()
        for line in content.split('\n'):
            if line.find(':') <= 0:
                continue
            chunks = line.split(':')
            projmap[chunks[0]] = chunks[1].strip()

allto = email.utils.getaddresses([str(x) for x in patch.msg.get_all('to', [])])
        allto += email.utils.getaddresses([str(x) for x in patch.msg.get_all('cc', [])])
        listarc = patch.msg.get_all('list-archive', [])
        for entry in allto:
            if entry[1] in projmap:
                projurl = 'https://lore.kernel.org/%s/' % projmap[entry[1]]
                # Make sure we don't re-query the same project we just used
                reused = False
                for arcurl in listarc:
                    if projurl in arcurl:
                        reused = True
                        break
                if reused:
                    continue
                # Try to backfill from that project
                tmp_mbox = mkstemp('b4-backfill-mbox')[1]
                get_pi_thread_by_msgid(patch.msgid, tmp_mbox, useproject=projmap[entry[1]])
                mbx = mailbox.mbox(tmp_mbox)
                was = len(self.msgid_map)
                for msg in mbx:
                    self.add_message(msg)
                mbx.close()
                os.unlink(tmp_mbox)
                if len(self.msgid_map) > was:
                    logger.info('Loaded %s messages from %s', len(self.msgid_map)-was, projurl)
                if self.series[revision].complete:
                    logger.info('Successfully backfilled missing patches')
                    break

def get_series(self, revision=None, sloppytrailers=False, backfill=True):
        if revision is None:
            if not len(self.series):
                return None
            # Use the highest revision
            revision = max(self.series.keys())
        elif revision not in self.series.keys():
            return None

lser = self.series[revision]

# Is it empty?
        empty = True
        for lmsg in lser.patches:
            if lmsg is not None:
                empty = False
                break
        if empty:
            logger.critical('All patches in series v%s are missing.', lser.revision)
            return None

if not lser.complete and backfill:
            self.backfill(revision)

# Grab our cover letter if we have one
        if revision in self.covers.keys():
            lser.add_patch(self.covers[revision])
            lser.has_cover = True
        else:
            # Let's find the first patch with an in-reply-to and see if that
            # is our cover letter
            for member in lser.patches:
                if member is not None and member.in_reply_to is not None:
                    potential = self.get_by_msgid(member.in_reply_to)
                    if potential is not None and potential.has_diffstat and not potential.has_diff:
                        # This is *probably* the cover letter
                        lser.patches[0] = potential
                        lser.has_cover = True
                        break

# Do we have any follow-ups?
        for fmsg in self.followups:
            logger.debug('Analyzing follow-up: %s (%s)', fmsg.full_subject, fmsg.fromemail)
            # If there are no trailers in this one, ignore it
            if not len(fmsg.trailers):
                logger.debug('  no trailers found, skipping')
                continue
            # Go up through the follow-ups and tally up trailers until
            # we either run out of in-reply-tos, or we find a patch in
            # one of our series
            if fmsg.in_reply_to is None:
                # Check if there's something matching in References
                refs = fmsg.msg.get('References', '')
                pmsg = None
                for ref in refs.split():
                    refid = ref.strip('<>')
                    if refid in self.msgid_map and refid != fmsg.msgid:
                        pmsg = self.msgid_map[refid]
                        break
                if pmsg is None:
                    # Can't find the message we're replying to here
                    continue
            elif fmsg.in_reply_to in self.msgid_map:
                pmsg = self.msgid_map[fmsg.in_reply_to]
            else:
                logger.debug('  missing message, skipping: %s', fmsg.in_reply_to)
                continue

trailers, mismatches = fmsg.get_trailers(sloppy=sloppytrailers)
            for trailer in mismatches:
                lser.trailer_mismatches.add((trailer[0], trailer[1], fmsg.fromname, fmsg.fromemail))
            lvl = 1
            while True:
                logger.debug('%sParent: %s', ' ' * lvl, pmsg.full_subject)
                logger.debug('%sTrailers:', ' ' * lvl)
                for trailer in trailers:
                    logger.debug('%s%s: %s', ' ' * (lvl+1), trailer[0], trailer[1])
                if pmsg.has_diff and not pmsg.reply:
                    # We found the patch for these trailers
                    if pmsg.revision != revision:
                        # add this into our trailer map to carry over trailers from
                        # previous revisions to current revision if patch/metadata did
                        # not change
                        pmsg.load_hashes()
                        if pmsg.attestation:
                            attid = pmsg.attestation.attid
                            if attid not in self.trailer_map:
                                self.trailer_map[attid] = list()
                            self.trailer_map[attid] += trailers
                    pmsg.followup_trailers += trailers
                    break
                if not pmsg.reply:
                    # Could be a cover letter
                    pmsg.followup_trailers += trailers
                    break
                if pmsg.in_reply_to and pmsg.in_reply_to in self.msgid_map:
                    lvl += 1
                    trailers += pmsg.trailers
                    pmsg = self.msgid_map[pmsg.in_reply_to]
                    continue
                break

# Carry over trailers from previous series if patch/metadata did not change
        for lmsg in lser.patches:
            if lmsg is None or lmsg.attestation is None:
                continue
            lmsg.load_hashes()
            if lmsg.attestation.attid in self.trailer_map:
                lmsg.followup_trailers += self.trailer_map[lmsg.attestation.attid]

return lser

def add_message(self, msg):
        msgid = LoreMessage.get_clean_msgid(msg)
        if msgid in self.msgid_map:
            logger.debug('Already have a message with this msgid, skipping %s', msgid)
            return

lmsg = LoreMessage(msg)
        logger.debug('Looking at: %s', lmsg.full_subject)
        self.msgid_map[lmsg.msgid] = lmsg

if lmsg.reply:
            # We'll figure out where this belongs later
            logger.debug('  adding to followups')
            self.followups.append(lmsg)
            return

if lmsg.counter == 0 and (not lmsg.counters_inferred or lmsg.has_diffstat):
            # Cover letter
            # Add it to covers -- we'll deal with them later
            logger.debug('  adding as v%s cover letter', lmsg.revision)
            self.covers[lmsg.revision] = lmsg
            return

if lmsg.has_diff:
            if lmsg.revision not in self.series:
                if lmsg.revision_inferred and lmsg.in_reply_to:
                    # We have an inferred revision here.
                    # Do we have an upthread cover letter that specifies a revision?
                    irt = self.get_by_msgid(lmsg.in_reply_to)
                    if irt is not None and irt.has_diffstat and not irt.has_diff:
                        # Yes, this is very likely our cover letter
                        logger.debug('  fixed revision to v%s', irt.revision)
                        lmsg.revision = irt.revision
                    # alternatively, see if upthread is patch 1
                    elif lmsg.counter > 0 and irt is not None and irt.has_diff and irt.counter == 1:
                        logger.debug('  fixed revision to v%s', irt.revision)
                        lmsg.revision = irt.revision

# Run our check again
            if lmsg.revision not in self.series:
                self.series[lmsg.revision] = LoreSeries(lmsg.revision, lmsg.expected)
                if len(self.series) > 1:
                    logger.debug('Found new series v%s', lmsg.revision)

# Attempt to auto-number series from the same author who did not bother
            # to set v2, v3, etc in the patch revision
            if (lmsg.counter == 1 and lmsg.counters_inferred
                    and not lmsg.reply and lmsg.lsubject.patch and not lmsg.lsubject.resend):
                omsg = self.series[lmsg.revision].patches[lmsg.counter]
                if (omsg is not None and omsg.counters_inferred and lmsg.fromemail == omsg.fromemail
                        and omsg.date < lmsg.date):
                    lmsg.revision = len(self.series) + 1
                    self.series[lmsg.revision] = LoreSeries(lmsg.revision, lmsg.expected)
                    logger.info('Assuming new revision: v%s (%s)', lmsg.revision, lmsg.full_subject)
            logger.debug('  adding as patch')
            self.series[lmsg.revision].add_patch(lmsg)
            return

logger.debug('  adding to unknowns')
        self.unknowns.append(lmsg)

class LoreSeries:
    def __init__(self, revision, expected):
        self.revision = revision
        self.expected = expected
        self.patches = [None] * (expected+1)
        self.followups = list()
        self.trailer_mismatches = set()
        self.complete = False
        self.has_cover = False
        self.subject = '(untitled)'

def __repr__(self):
        out = list()
        out.append('- Series: [v%s] %s' % (self.revision, self.subject))
        out.append('  revision: %s' % self.revision)
        out.append('  expected: %s' % self.expected)
        out.append('  complete: %s' % self.complete)
        out.append('  has_cover: %s' % self.has_cover)
        out.append('  patches:')
        at = 0
        for member in self.patches:
            if member is not None:
                out.append('    [%s/%s] %s' % (at, self.expected, member.subject))
                if member.followup_trailers:
                    out.append('       Add: %s' % ', '.join(member.followup_trailers))
            else:
                out.append('    [%s/%s] MISSING' % (at, self.expected))
            at += 1

return '\n'.join(out)

def add_patch(self, lmsg):
        while len(self.patches) < lmsg.expected + 1:
            self.patches.append(None)
        self.expected = lmsg.expected
        if self.patches[lmsg.counter] is not None:
            # Okay, weird, is the one in there a reply?
            omsg = self.patches[lmsg.counter]
            if omsg.reply or (omsg.counters_inferred and not lmsg.counters_inferred):
                # Replace that one with this one
                logger.debug('  replacing existing: %s', omsg.subject)
                self.patches[lmsg.counter] = lmsg
        else:
            self.patches[lmsg.counter] = lmsg
        self.complete = not (None in self.patches[1:])
        if self.patches[0] is not None:
            # noinspection PyUnresolvedReferences
            self.subject = self.patches[0].subject
        elif self.patches[1] is not None:
            # noinspection PyUnresolvedReferences
            self.subject = self.patches[1].subject

def get_slug(self, extended=False):
        # Find the first non-None entry
        lmsg = None
        for lmsg in self.patches:
            if lmsg is not None:
                break

if lmsg is None:
            return 'undefined'

prefix = lmsg.date.strftime('%Y%m%d')
        authorline = email.utils.getaddresses([str(x) for x in lmsg.msg.get_all('from', [])])[0]
        if extended:
            local = authorline[1].split('@')[0]
            unsafe = '%s_%s_%s' % (prefix, local, lmsg.subject)
            slug = re.sub(r'\W+', '_', unsafe).strip('_').lower()
        else:
            author = re.sub(r'\W+', '_', authorline[1]).strip('_').lower()
            slug = '%s_%s' % (prefix, author)

if self.revision != 1:
            slug = 'v%s_%s' % (self.revision, slug)

return slug[:100]

def save_am_mbox(self, mbx, noaddtrailers=False, covertrailers=False, trailer_order=None, addmysob=False,
                     addlink=False, linkmask=None, cherrypick=None, copyccs=False):

usercfg = get_user_config()
        config = get_main_config()

if addmysob:
            if 'name' not in usercfg or 'email' not in usercfg:
                logger.critical('WARNING: Unable to add your Signed-off-by: git returned no user.name or user.email')
                addmysob = False

attdata = [(None, None)] * len(self.patches[1:])
        attpolicy = config['attestation-policy']

if config['attestation-checkmarks'] == 'fancy':
            attpass = PASS_FANCY
            attfail = FAIL_FANCY
            attweak = WEAK_FANCY
        else:
            attpass = PASS_SIMPLE
            attfail = FAIL_SIMPLE
            attweak = WEAK_SIMPLE

at = 1
        atterrors = list()
        for lmsg in self.patches[1:]:
            if cherrypick is not None:
                if at not in cherrypick:
                    at += 1
                    logger.debug('  skipped: [%s/%s] (not in cherrypick)', at, self.expected)
                    continue
                if lmsg is None:
                    logger.critical('CRITICAL: [%s/%s] is missing, cannot cherrypick', at, self.expected)
                    raise KeyError('Cherrypick not in series')

if lmsg is not None:
                if self.has_cover and covertrailers and self.patches[0].followup_trailers:  # noqa
                    lmsg.followup_trailers += self.patches[0].followup_trailers  # noqa
                if addmysob:
                    lmsg.followup_trailers.append(('Signed-off-by',
                                                   '%s <%s>' % (usercfg['name'], usercfg['email']), None, None))
                if addlink:
                    lmsg.followup_trailers.append(('Link', linkmask % lmsg.msgid, None, None))

if attpolicy != 'off':
                    lmsg.load_hashes()
                    latt = lmsg.attestation
                    if latt and latt.validate(lmsg.msg):
                        if latt.lsig.attestor and latt.lsig.attestor.mode == 'domain':
                            logger.info('  %s %s', attweak, lmsg.full_subject)
                            attdata[at-1] = (latt.lsig.attestor.get_trailer(lmsg.fromemail), attweak) # noqa
                        else:
                            logger.info('  %s %s', attpass, lmsg.full_subject)
                            attdata[at-1] = (latt.lsig.attestor.get_trailer(lmsg.fromemail), attpass) # noqa
                    else:
                        if latt and latt.lsig and attpolicy in ('softfail', 'hardfail'):
                            logger.info('  %s %s', attfail, lmsg.full_subject)
                            if latt and latt.lsig and latt.lsig.attestor and latt.lsig.attestor.mode == 'domain':
                                atterrors.append('Failed %s attestation' % latt.lsig.attestor.get_trailer())
                            elif latt and latt.lsig and latt.lsig.attestor:
                                failed = list()
                                if not latt.pv:
                                    failed.append('patch content')
                                if not latt.mv:
                                    failed.append('commit message')
                                if not latt.iv:
                                    failed.append('patch metadata')
                                atterrors.append('Patch %s/%s failed attestation (%s)' % (at, lmsg.expected,
                                                                                          ', '.join(failed)))
                        else:
                            logger.info('  %s', lmsg.full_subject)
                else:
                    logger.info('  %s', lmsg.full_subject)

add_trailers = True
                if noaddtrailers:
                    add_trailers = False
                msg = lmsg.get_am_message(add_trailers=add_trailers, trailer_order=trailer_order, copyccs=copyccs)
                # Pass a policy that avoids most legacy encoding horrors
                mbx.add(msg.as_bytes(policy=emlpolicy))
            else:
                logger.error('  ERROR: missing [%s/%s]!', at, self.expected)
            at += 1

if attpolicy == 'off':
            return mbx

failed = (None, None) in attdata
        if not failed:
            logger.info('  ---')
            for trailer, attmode in set(attdata):
                logger.info('  %s Attestation-by: %s', attmode, trailer)
            return mbx
        elif not can_dkim_verify and config.get('attestation-check-dkim') == 'yes':
            logger.info('  ---')
            logger.info('  NOTE: install dkimpy for DKIM signature verification')

errors = set(atterrors)
        for attdoc in ATTESTATIONS:
            errors.update(attdoc.errors)

if errors:
            logger.critical('  ---')
            logger.critical('  Attestation is available, but did not succeed:')
            for error in errors:
                logger.critical('    %s %s', attfail, error)

if attpolicy == 'hardfail':
            import sys
            sys.exit(128)

return mbx

def check_applies_clean(self, gitdir, when=None):
        # Go through indexes and see if this series should apply cleanly
        mismatches = 0
        seenfiles = set()
        for lmsg in self.patches[1:]:
            if lmsg is None or lmsg.blob_indexes is None:
                continue
            for fn, bh in lmsg.blob_indexes:
                if fn in seenfiles:
                    # if we have seen this file once already, then it's a repeat patch
                    # and it's no longer going to match current hash
                    continue
                seenfiles.add(fn)
                if set(bh) == {'0'}:
                    # New file, will for sure apply clean
                    continue
                fullpath = os.path.join(gitdir, fn)
                if when is None:
                    if not os.path.exists(fullpath):
                        mismatches += 1
                        continue
                    cmdargs = ['hash-object', fullpath]
                    ecode, out = git_run_command(None, cmdargs)
                else:
                    gitdir = os.path.join(gitdir, '.git')
                    logger.debug('Checking hash on %s:%s', when, fn)
                    # XXX: We should probably pipe the two commands instead of reading into memory,
                    #      so something to consider for the future
                    ecode, out = git_run_command(gitdir, ['show', f'{when}:{fn}'])
                    if ecode > 0:
                        # Couldn't get this file, continue
                        logger.debug('Could not look up %s:%s', when, fn)
                        mismatches += 1
                        continue
                    cmdargs = ['hash-object', '--stdin']
                    ecode, out = git_run_command(None, cmdargs, stdin=out.encode())
                if ecode == 0:
                    if out.find(bh) != 0:
                        logger.debug('%s hash: %s (expected: %s)', fn, out.strip(), bh)
                        mismatches += 1
                    else:
                        logger.debug('%s hash: matched', fn)

return len(seenfiles), mismatches

def make_fake_am_range(self, gitdir):
        start_commit = end_commit = None
        # Use the msgid of the first non-None patch in the series
        msgid = None
        for lmsg in self.patches:
            if lmsg is not None:
                msgid = lmsg.msgid
                break
        if msgid is None:
            logger.critical('Cannot operate on an empty series')
            return None, None
        cachedata = get_cache(msgid, suffix='fakeam')
        if cachedata:
            stalecache = False
            chunks = cachedata.strip().split()
            if len(chunks) == 2:
                start_commit, end_commit = chunks
            else:
                stalecache = True
            if start_commit is not None and end_commit is not None:
                # Make sure they are still there
                ecode, out = git_run_command(gitdir, ['cat-file', '-e', start_commit])
                if ecode > 0:
                    stalecache = True
                else:
                    ecode, out = git_run_command(gitdir, ['cat-file', '-e', end_commit])
                    if ecode > 0:
                        stalecache = True
                    else:
                        logger.debug('Using previously generated range')
                        return start_commit, end_commit

if stalecache:
                logger.debug('Stale cache for [v%s] %s', self.revision, self.subject)
                save_cache(None, msgid, suffix='fakeam')

logger.info('Preparing fake-am for v%s: %s', self.revision, self.subject)
        with git_temp_worktree(gitdir):
            # We are in a temporary chdir at this time, so writing to a known file should be safe
            mbxf = '.__git-am__'
            mbx = mailbox.mbox(mbxf)
            # Logic largely borrowed from gj_tools
            seenfiles = set()
            for lmsg in self.patches[1:]:
                if lmsg is None:
                    logger.critical('ERROR: v%s series incomplete; unable to create a fake-am range', self.revision)
                    return None, None
                logger.debug('Looking at %s', lmsg.full_subject)
                lmsg.load_hashes()
                if not lmsg.blob_indexes:
                    logger.critical('ERROR: some patches do not have indexes')
                    logger.critical('       unable to create a fake-am range')
                    return None, None
                for fn, fi in lmsg.blob_indexes:
                    if fn in seenfiles:
                        # We already processed this file, so this blob won't match
                        continue
                    seenfiles.add(fn)
                    if set(fi) == {'0'}:
                        # New file creation, nothing to do here
                        logger.debug('  New file: %s', fn)
                        continue
                    # Try to grab full ref_id of this hash
                    ecode, out = git_run_command(gitdir, ['rev-parse', fi])
                    if ecode > 0:
                        logger.critical('  ERROR: Could not find matching blob for %s (%s)', fn, fi)
                        logger.critical('         If you know on which tree this patchset is based,')
                        logger.critical('         add it as a remote and perform "git remote update"')
                        logger.critical('         in order to fetch the missing objects.')
                        return None, None
                    logger.debug('  Found matching blob for: %s', fn)
                    fullref = out.strip()
                    gitargs = ['update-index', '--add', '--cacheinfo', f'0644,{fullref},{fn}']
                    ecode, out = git_run_command(None, gitargs)
                    if ecode > 0:
                        logger.critical('  ERROR: Could not run update-index for %s (%s)', fn, fullref)
                        return None, None
                mbx.add(lmsg.msg.as_string(policy=emlpolicy).encode('utf-8'))

mbx.close()
            ecode, out = git_run_command(None, ['write-tree'])
            if ecode > 0:
                logger.critical('ERROR: Could not write fake-am tree')
                return None, None
            treeid = out.strip()
            # At this point we have a worktree with files that should cleanly receive a git am
            gitargs = ['commit-tree', treeid + '^{tree}', '-F', '-']
            ecode, out = git_run_command(None, gitargs, stdin='Initial fake commit'.encode('utf-8'))
            if ecode > 0:
                logger.critical('ERROR: Could not commit-tree')
                return None, None
            start_commit = out.strip()
            git_run_command(None, ['reset', '--hard', start_commit])
            ecode, out = git_run_command(None, ['am', mbxf])
            if ecode > 0:
                logger.critical('ERROR: Could not fake-am version %s', self.revision)
                return None, None
            ecode, out = git_run_command(None, ['rev-parse', 'HEAD'])
            end_commit = out.strip()
            logger.info('  range: %.12s..%.12s', start_commit, end_commit)

logger.debug('Saving into cache:')
        logger.debug('    %s..%s', start_commit, end_commit)
        save_cache(f'{start_commit} {end_commit}\n', msgid, suffix='fakeam')

return start_commit, end_commit

def save_cover(self, outfile):
        # noinspection PyUnresolvedReferences
        cover_msg = self.patches[0].get_am_message(add_trailers=False, trailer_order=None)
        with open(outfile, 'w') as fh:
            fh.write(cover_msg.as_string(policy=emlpolicy))
        logger.critical('Cover: %s', outfile)

class LoreMessage:
    def __init__(self, msg):
        self.msg = msg
        self.msgid = None

# Subject-based info
        self.lsubject = None
        self.full_subject = None
        self.subject = None
        self.reply = False
        self.revision = 1
        self.counter = 1
        self.expected = 1
        self.revision_inferred = True
        self.counters_inferred = True

# Header-based info
        self.in_reply_to = None
        self.fromname = None
        self.fromemail = None
        self.date = None

# Body and body-based info
        self.body = None
        self.charset = 'utf-8'
        self.has_diff = False
        self.has_diffstat = False
        self.trailers = list()
        self.followup_trailers = list()

# These are populated by pr
        self.pr_base_commit = None
        self.pr_repo = None
        self.pr_ref = None
        self.pr_tip_commit = None
        self.pr_remote_tip_commit = None

self.attestation = None
        # Patchwork hash
        self.pwhash = None
        # Git patch-id
        self.git_patch_id = None
        # Blob indexes
        self.blob_indexes = None

self.msgid = LoreMessage.get_clean_msgid(self.msg)
        self.lsubject = LoreSubject(msg['Subject'])
        # Copy them into this object for convenience
        self.full_subject = self.lsubject.full_subject
        self.subject = self.lsubject.subject
        self.reply = self.lsubject.reply
        self.revision = self.lsubject.revision
        self.counter = self.lsubject.counter
        self.expected = self.lsubject.expected
        self.revision_inferred = self.lsubject.revision_inferred
        self.counters_inferred = self.lsubject.counters_inferred

# Handle [PATCH 6/5]
        if self.counter > self.expected:
            self.expected = self.counter

self.in_reply_to = LoreMessage.get_clean_msgid(self.msg, header='In-Reply-To')

try:
            fromdata = email.utils.getaddresses([LoreMessage.clean_header(str(x))
                                                 for x in self.msg.get_all('from', [])])[0]
            self.fromname = fromdata[0]
            self.fromemail = fromdata[1]
            if not len(self.fromname.strip()):
                self.fromname = self.fromemail
        except IndexError:
            pass

msgdate = self.msg.get('Date')
        if msgdate:
            self.date = email.utils.parsedate_to_datetime(str(msgdate))
        else:
            # An email without a Date: field?
            self.date = datetime.datetime.now()

# walk until we find the first text/plain part
        mcharset = self.msg.get_content_charset()
        if not mcharset:
            mcharset = 'utf-8'
        self.charset = mcharset

for part in msg.walk():
            cte = part.get_content_type()
            if cte.find('/plain') < 0 and cte.find('/x-patch') < 0:
                continue
            payload = part.get_payload(decode=True)
            if payload is None:
                continue
            pcharset = part.get_content_charset()
            if not pcharset:
                pcharset = mcharset
            try:
                payload = payload.decode(pcharset, errors='replace')
                self.charset = pcharset
            except LookupError:
                # what kind of encoding is that?
                # Whatever, we'll use utf-8 and hope for the best
                payload = payload.decode('utf-8', errors='replace')
                part.set_param('charset', 'utf-8')
                self.charset = 'utf-8'
            if self.body is None:
                self.body = payload
                continue
            # If we already found a body, but we now find something that contains a diff,
            # then we prefer this part
            if diffre.search(payload):
                self.body = payload

if self.body is None:
            # Woah, we didn't find any usable parts
            logger.debug('  No plain or patch parts found in message')
            logger.info('  Not plaintext: %s', self.full_subject)
            return

if diffstatre.search(self.body):
            self.has_diffstat = True
        if diffre.search(self.body):
            self.has_diff = True

# We only pay attention to trailers that are sent in reply
        if self.reply:
            trailers, others = LoreMessage.find_trailers(self.body, followup=True)
            for trailer in trailers:
                # These are commonly part of patch/commit metadata
                badtrailers = ('from', 'author', 'cc', 'to')
                if trailer[0].lower() not in badtrailers:
                    self.trailers.append(trailer)

def get_trailers(self, sloppy=False):
        mismatches = set()
        if sloppy:
            return self.trailers, mismatches

trailers = list()
        for tname, tvalue, extdata in self.trailers:
            if tname.lower() in ('fixes',):
                trailers.append((tname, tvalue, extdata, self))
                continue

tmatch = False
            namedata = email.utils.getaddresses([tvalue])[0]
            tfrom = re.sub(r'\+[^@]+@', '@', namedata[1].lower())
            hfrom = re.sub(r'\+[^@]+@', '@', self.fromemail.lower())
            tlname = namedata[0].lower()
            hlname = self.fromname.lower()
            tchunks = tfrom.split('@')
            hchunks = hfrom.split('@')
            if tfrom == hfrom:
                logger.debug('  trailer exact email match')
                tmatch = True
            # See if domain part of one of the addresses is a subset of the other one,
            # which should match cases like @linux.intel.com and @intel.com
            elif (len(tchunks) == 2 and len(hchunks) == 2
                  and tchunks[0] == hchunks[0]
                  and (tchunks[1].find(hchunks[1]) >= 0 or hchunks[1].find(tchunks[1]) >= 0)):
                logger.debug('  trailer fuzzy email match')
                tmatch = True
            # Does the name match, at least?
            elif tlname == hlname:
                logger.debug('  trailer exact name match')
                tmatch = True
            # Finally, see if the header From has a comma in it and try to find all
            # parts in the trailer name
            elif hlname.find(',') > 0:
                nmatch = True
                for nchunk in hlname.split(','):
                    if hlname.find(nchunk.strip()) < 0:
                        nmatch = False
                        break
                if nmatch:
                    logger.debug('  trailer fuzzy name match')
                    tmatch = True
            if tmatch:
                trailers.append((tname, tvalue, extdata, self))
            else:
                mismatches.add((tname, tvalue, extdata, self))

return trailers, mismatches

def __repr__(self):
        out = list()
        out.append('msgid: %s' % self.msgid)
        out.append(str(self.lsubject))

out.append('  fromname: %s' % self.fromname)
        out.append('  fromemail: %s' % self.fromemail)
        out.append('  date: %s' % str(self.date))
        out.append('  in_reply_to: %s' % self.in_reply_to)

# Header-based info
        out.append('  --- begin body ---')
        for line in self.body.split('\n'):
            out.append('  |%s' % line)
        out.append('  --- end body ---')

# Body and body-based info
        out.append('  has_diff: %s' % self.has_diff)
        out.append('  has_diffstat: %s' % self.has_diffstat)
        out.append('  --- begin my trailers ---')
        for trailer in self.trailers:
            out.append('  |%s' % str(trailer))
        out.append('  --- begin followup trailers ---')
        for trailer in self.followup_trailers:
            out.append('  |%s' % str(trailer))
        out.append('  --- end trailers ---')

return '\n'.join(out)

@staticmethod
    def clean_header(hdrval):
        if hdrval is None:
            return ''

decoded = ''
        for hstr, hcs in email.header.decode_header(hdrval):
            if hcs is None:
                hcs = 'utf-8'
            try:
                decoded += hstr.decode(hcs)
            except LookupError:
                # Try as utf-u
                decoded += hstr.decode('utf-8', errors='replace')
            except (UnicodeDecodeError, AttributeError):
                decoded += hstr
        new_hdrval = re.sub(r'\n?\s+', ' ', decoded)
        return new_hdrval.strip()

@staticmethod
    def get_clean_msgid(msg, header='Message-Id'):
        msgid = None
        raw = msg.get(header)
        if raw:
            matches = re.search(r'<([^>]+)>', LoreMessage.clean_header(raw))
            if matches:
                msgid = matches.groups()[0]
        return msgid

@staticmethod
    def get_patchwork_hash(diff):
        # Make sure we just have the diff without any extraneous content.
        diff = LoreMessage.get_clean_diff(diff)
        """Generate a hash from a diff. Lifted verbatim from patchwork."""

prefixes = ['-', '+', ' ']
        hashed = hashlib.sha1()

for line in diff.split('\n'):
            if len(line) <= 0:
                continue

hunk_match = HUNK_RE.match(line)
            filename_match = FILENAME_RE.match(line)

if filename_match:
                # normalise -p1 top-directories
                if filename_match.group(1) == '---':
                    filename = 'a/'
                else:
                    filename = 'b/'
                filename += '/'.join(filename_match.group(2).split('/')[1:])

line = filename_match.group(1) + ' ' + filename
            elif hunk_match:
                # remove line numbers, but leave line counts
                def fn(x):
                    if not x:
                        return 1
                    return int(x)

line_nos = list(map(fn, hunk_match.groups()))
                line = '@@ -%d +%d @@' % tuple(line_nos)
            elif line[0] in prefixes:
                # if we have a +, - or context line, leave as-is
                pass
            else:
                # other lines are ignored
                continue

hashed.update((line + '\n').encode('utf-8'))

return hashed.hexdigest()

@staticmethod
    def get_indexes(diff):
        indexes = set()
        curfile = None
        for line in diff.split('\n'):
            if line.find('diff ') != 0 and line.find('index ') != 0:
                continue
            matches = re.search(r'^diff\s+--git\s+\w/(.*)\s+\w/(.*)$', line)
            if matches and matches.groups()[0] == matches.groups()[1]:
                curfile = matches.groups()[0]
                continue
            matches = re.search(r'^index\s+([0-9a-f]+)\.\.[0-9a-f]+.*$', line)
            if matches and curfile is not None:
                indexes.add((curfile, matches.groups()[0]))
        return indexes

@staticmethod
    def get_clean_diff(diff):
        diff = diff.replace('\r', '')

# For keeping a buffer of lines preceding @@ ... @@
        buflines = list()
        difflines = ''

# Used for counting where we are in the patch
        pp = mm = 0
        inside_binary_chunk = False
        for line in diff.split('\n'):
            if not len(line):
                if inside_binary_chunk:
                    inside_binary_chunk = False
                    # add all buflines to difflines
                    difflines += '\n'.join(buflines) + '\n\n'
                    buflines = list()
                    continue
                buflines.append(line)
                continue
            elif inside_binary_chunk:
                buflines.append(line)
                continue
            # If line starts with 'index ' and previous line starts with 'deleted ', then
            # it's a file delete and therefore doesn't have a regular hunk.
            if line.find('index ') == 0 and len(buflines) > 1 and buflines[-1].find('deleted ') == 0:
                # add this and 2 preceding lines to difflines and reset buflines
                buflines.append(line)
                difflines += '\n'.join(buflines[-3:]) + '\n'
                buflines = list()
                continue
            if line.find('delta ') == 0 or line.find('literal ') == 0:
                # we are inside a binary patch
                inside_binary_chunk = True
                buflines.append(line)
                continue
            hunk_match = HUNK_RE.match(line)
            if hunk_match:
                # logger.debug('Crunching %s', line)
                mlines, plines = hunk_match.groups()
                try:
                    pp = int(plines)
                except TypeError:
                    pp = 1
                try:
                    mm = int(mlines)
                except TypeError:
                    mm = 1
                addlines = list()
                for bline in reversed(buflines):
                    # Go backward and add lines until we get to the start
                    # or encounter a blank line
                    if len(bline.strip()) == 0:
                        break
                    addlines.append(bline)
                if addlines:
                    difflines += '\n'.join(reversed(addlines)) + '\n'
                buflines = list()
                # Feed this line to the hasher
                difflines += line + '\n'
                continue
            if pp > 0 or mm > 0:
                # Inside the patch
                difflines += line + '\n'
                if line[0] in (' ', '-'):
                    mm -= 1
                if line[0] in (' ', '+'):
                    pp -= 1
                continue
            # Not anything we recognize, so stick into buflines
            buflines.append(line)
        return difflines

def load_hashes(self):
        if self.attestation is not None:
            return
        logger.debug('Calculating hashes for: %s', self.full_subject)
        # Calculate git-patch-id first
        cmdargs = ['patch-id', '--stable']
        msg = self.get_am_message(add_trailers=False)
        stdin = msg.as_string(policy=emlpolicy).encode()
        ecode, out = git_run_command(None, cmdargs, stdin)
        if ecode > 0:
            # Git doesn't think there's a patch there
            return
        fline = out.split('\n')[0]
        if len(fline) >= 40:
            self.git_patch_id = fline[:40]

msg_out = mkstemp()
        patch_out = mkstemp()
        cmdargs = ['mailinfo', '--encoding=UTF-8', msg_out[1], patch_out[1]]
        ecode, info = git_run_command(None, cmdargs, stdin)
        if ecode > 0:
            logger.debug('ERROR: Could not get mailinfo')
            return
        i = hashlib.sha256()
        m = hashlib.sha256()
        p = hashlib.sha256()

for line in info.split('\n'):
            # We don't use the "Date:" field because it is likely to be
            # mangled between when git-format-patch generates it and
            # when it is sent out by git-send-email (or other tools).
            if re.search(r'^(Author|Email|Subject):', line):
                i.update((line + '\n').encode())

with open(msg_out[1], 'rb') as mfh:
            msg = mfh.read()
            m.update(msg)
        os.unlink(msg_out[1])

with open(patch_out[1], 'rb') as pfh:
            patch = pfh.read().decode(self.charset, errors='replace')
            if len(patch.strip()):
                diff = LoreMessage.get_clean_diff(patch)
                p.update(diff.encode())
                self.pwhash = LoreMessage.get_patchwork_hash(patch)
                # Load the indexes, if we have them
                self.blob_indexes = LoreMessage.get_indexes(diff)
            else:
                p = None

os.unlink(patch_out[1])

if i and m and p:
            self.attestation = LoreAttestation(i, m, p)

@staticmethod
    def find_trailers(body, followup=False):
        headers = ('subject', 'date', 'from')
        nonperson = ('fixes', 'subject', 'date', 'link', 'buglink')
        # Fix some more common copypasta trailer wrapping
        # Fixes: abcd0123 (foo bar
        # baz quux)
        body = re.sub(r'^(\S+:\s+[0-9a-f]+\s+$[^)]+)\n([^\n]+$)', r'\1 \2', body, flags=re.M)
        # Signed-off-by: Long Name
        # <email.here@example.com>
        body = re.sub(r'^(\S+:\s+[^<]+)\n(<[^>]+>)$', r'\1 \2', body, flags=re.M)
        # Signed-off-by: Foo foo <foo@foo.com>
        # [for the thing that the thing is too long the thing that is
        # thing but thing]
        # (too false-positivey, commented out)
        # body = re.sub(r'^(\[[^]]+)\n([^]]+]$)', r'\1 \2', body, flags=re.M)
        trailers = list()
        others = list()
        was_trailer = False
        for line in body.split('\n'):
            line = line.strip('\r')
            matches = re.search(r'^(\w\S+):\s+(\S.*)', line, flags=re.I)
            if matches:
                groups = list(matches.groups())
                # We only accept headers if we haven't seen any non-trailer lines
                tname = groups[0].lower()
                if len(others) and tname in headers:
                    logger.debug('Ignoring %s (header after other content)', line)
                    continue
                if followup:
                    mperson = re.search(r'\S+@\S+\.\S+', groups[1])
                    if not mperson and tname not in nonperson:
                        logger.debug('Ignoring %s (not a recognized non-person trailer)', line)
                        continue
                was_trailer = True
                groups.append(None)
                trailers.append(groups)
                continue
            # Is it an extended info line, e.g.:
            # Signed-off-by: Foo Foo <foo@foo.com>
            # [for the foo bits]
            if len(line) > 2 and line[0] == '[' and line[-1] == ']' and was_trailer:
                trailers[-1][2] = line
                was_trailer = False
                continue
            was_trailer = False
            others.append(line)

return trailers, others

@staticmethod
    def get_body_parts(body):
        # remove any starting/trailing blank lines
        body = body.replace('\r', '')
        body = body.strip('\n')
        # Extra git-relevant headers, like From:, Subject:, Date:, etc
        githeaders = list()
        # commit message
        message = ''
        # everything below the ---
        basement = ''
        # conformant signature --\s\n
        signature = ''
        sparts = body.rsplit('\n-- \n', 1)
        if len(sparts) > 1:
            signature = sparts[1]
            body = sparts[0].rstrip('\n')

parts = re.split('^---\n', body, maxsplit=1, flags=re.M)
        if len(parts) == 2:
            basement = parts[1].rstrip('\n')
        elif body.find('\ndiff ') >= 0:
            parts = body.split('\ndiff ', 1)
            if len(parts) == 2:
                parts[1] = 'diff ' + parts[1]
            basement = parts[1].rstrip('\n')

mbody = parts[0].strip('\n')

# Split into paragraphs
        bpara = mbody.split('\n\n')

# Is every line of the first part in a header format?
        mparts = list()
        h, o = LoreMessage.find_trailers(bpara[0])
        if len(o):
            # Not everything was a header, so we don't treat it as headers
            mparts.append(bpara[0])
        else:
            githeaders = h

# Any lines of the last part match the header format?
        trailers, nlines = LoreMessage.find_trailers(bpara[-1])

if len(bpara) == 1:
            if githeaders == trailers:
                # This is a message that consists of just trailers?
                githeaders = list()
            if nlines:
                message = '\n'.join(nlines)
            return githeaders, message, trailers, basement, signature

# Add all parts between first and last to mparts
        if len(bpara) > 2:
            mparts += bpara[1:-1]

if len(nlines):
            # Add them as the last part
            mparts.append('\n'.join(nlines))

message = '\n\n'.join(mparts)

return githeaders, message, trailers, basement, signature

def fix_trailers(self, trailer_order=None, copyccs=False):
        config = get_main_config()
        attpolicy = config['attestation-policy']

if config['attestation-checkmarks'] == 'fancy':
            attfail = FAIL_FANCY
            attweak = WEAK_FANCY
        else:
            attfail = FAIL_SIMPLE
            attweak = WEAK_SIMPLE

bheaders, message, btrailers, basement, signature = LoreMessage.get_body_parts(self.body)
        # Now we add mix-in trailers
        trailers = btrailers + self.followup_trailers

if copyccs:
            alldests = email.utils.getaddresses([str(x) for x in self.msg.get_all('to', [])])
            alldests += email.utils.getaddresses([str(x) for x in self.msg.get_all('cc', [])])
            # Sort by domain name, then local
            alldests.sort(key=lambda x: x[1].find('@') > 0 and x[1].split('@')[1] + x[1].split('@')[0] or x[1])
            for pair in alldests:
                found = False
                for ftr in trailers:
                    if ftr[1].lower().find(pair[1].lower()) >= 0:
                        # already present
                        found = True
                        break

if not found:
                    if len(pair[0]):
                        trailers.append(('Cc', f'{pair[0]} <{pair[1]}>', None, None))  # noqa
                    else:
                        trailers.append(('Cc', pair[1], None, None))  # noqa

fixtrailers = list()
        if trailer_order is None:
            trailer_order = DEFAULT_TRAILER_ORDER
        elif trailer_order in ('preserve', '_preserve_'):
            trailer_order = '*'
        for trailermatch in trailer_order:
            for trailer in trailers:
                if list(trailer[:3]) in fixtrailers:
                    # Dupe
                    continue
                if fnmatch.fnmatch(trailer[0].lower(), trailermatch.strip()):
                    fixtrailers.append(list(trailer[:3]))
                    if trailer[:3] not in btrailers:
                        extra = ''
                        if can_dkim_verify and config.get('attestation-check-dkim') == 'yes' and attpolicy != 'off':
                            if len(trailer) > 3 and trailer[3] is not None:
                                fmsg = trailer[3]
                                attsig = LoreAttestationSignatureDKIM(fmsg.msg)  # noqa
                                if attsig.present:
                                    if attsig.passing:
                                        extra = ' (%s %s)' % (attweak, attsig.attestor.get_trailer())
                                    elif attpolicy in ('softfail', 'hardfail'):
                                        extra = ' (%s %s)' % (attfail, attsig.attestor.get_trailer())
                        logger.info('    + %s: %s%s', trailer[0], trailer[1], extra)
                    else:
                        logger.debug('    . %s: %s', trailer[0], trailer[1])

# Reconstitute the message
        self.body = ''
        if bheaders:
            for bheader in bheaders:
                # There is no [extdata] in git headers, so we ignore bheader[2]
                self.body += '%s: %s\n' % (bheader[0], bheader[1])
            self.body += '\n'

if len(message):
            self.body += message + '\n'
            if len(fixtrailers):
                self.body += '\n'

if len(fixtrailers):
            for trailer in fixtrailers:
                self.body += '%s: %s\n' % (trailer[0], trailer[1])
                if trailer[2]:
                    self.body += '%s\n' % trailer[2]
        if len(basement):
            self.body += '---\n'
            self.body += basement
            self.body += '\n'
        if len(signature):
            self.body += '-- \n'
            self.body += signature
            self.body += '\n'

def get_am_message(self, add_trailers=True, trailer_order=None, copyccs=False):
        if add_trailers:
            self.fix_trailers(trailer_order=trailer_order, copyccs=copyccs)
        am_body = self.body
        am_msg = email.message.EmailMessage()
        am_msg.set_payload(am_body.encode('utf-8'))
        # Clean up headers
        for hdrname, hdrval in self.msg.items():
            lhdrname = hdrname.lower()
            wanthdr = False
            for hdrmatch in WANTHDRS:
                if fnmatch.fnmatch(lhdrname, hdrmatch):
                    wanthdr = True
                    break
            if wanthdr:
                new_hdrval = LoreMessage.clean_header(hdrval)
                # noinspection PyBroadException
                try:
                    am_msg.add_header(hdrname, new_hdrval)
                except:
                    # A broad except to handle any potential weird header conditions
                    pass
        am_msg.set_charset('utf-8')
        return am_msg

class LoreSubject:
    def __init__(self, subject):
        # Subject-based info
        self.full_subject = None
        self.subject = None
        self.reply = False
        self.resend = False
        self.patch = False
        self.rfc = False
        self.revision = 1
        self.counter = 1
        self.expected = 1
        self.revision_inferred = True
        self.counters_inferred = True
        self.prefixes = list()

subject = re.sub(r'\s+', ' ', LoreMessage.clean_header(subject)).strip()
        # Remove any leading [] that don't have "patch", "resend" or "rfc" in them
        while True:
            oldsubj = subject
            subject = re.sub(r'^\s*\[[^]]*]\s*(\[[^]]*(:?patch|resend|rfc).*)', r'\1', subject, flags=re.IGNORECASE)
            if oldsubj == subject:
                break

# Remove any brackets inside brackets
        while True:
            oldsubj = subject
            subject = re.sub(r'^\s*\[([^]]*)\[([^\[\]]*)]', r'[\1\2]', subject)
            subject = re.sub(r'^\s*\[([^]]*)]([^\[\]]*)]', r'[\1\2]', subject)
            if oldsubj == subject:
                break

self.full_subject = subject
        # Is it a reply?
        if re.search(r'^(Re|Aw|Fwd):', subject, re.I) or re.search(r'^\w{2,3}:\s*\[', subject):
            self.reply = True
            subject = re.sub(r'^\w+:\s*\[', '[', subject)

# Fix [PATCHv3] to be properly [PATCH v3]
        subject = re.sub(r'^\[\s*(patch)(v\d+)(.*)', r'[\1 \2\3', subject, flags=re.I)

# Find all [foo] in the title
        while subject.find('[') == 0:
            matches = re.search(r'^\[([^]]*)]', subject)
            if not matches:
                break
            for chunk in matches.groups()[0].split():
                # Remove any trailing commas or semicolons
                chunk = chunk.strip(',;')
                if re.search(r'^\d{1,3}/\d{1,3}$', chunk):
                    counters = chunk.split('/')
                    self.counter = int(counters[0])
                    self.expected = int(counters[1])
                    self.counters_inferred = False
                elif re.search(r'^v\d+$', chunk, re.IGNORECASE):
                    self.revision = int(chunk[1:])
                    self.revision_inferred = False
                elif chunk.lower().find('rfc') == 0:
                    self.rfc = True
                elif chunk.lower().find('resend') == 0:
                    self.resend = True
                elif chunk.lower().find('patch') == 0:
                    self.patch = True
                self.prefixes.append(chunk)
            subject = re.sub(r'^\s*\[[^]]*]\s*', '', subject)
        self.subject = subject

def __repr__(self):
        out = list()
        out.append('  full_subject: %s' % self.full_subject)
        out.append('  subject: %s' % self.subject)
        out.append('  reply: %s' % self.reply)
        out.append('  resend: %s' % self.resend)
        out.append('  patch: %s' % self.patch)
        out.append('  rfc: %s' % self.rfc)
        out.append('  revision: %s' % self.revision)
        out.append('  revision_inferred: %s' % self.revision_inferred)
        out.append('  counter: %s' % self.counter)
        out.append('  expected: %s' % self.expected)
        out.append('  counters_inferred: %s' % self.counters_inferred)
        out.append('  prefixes: %s' % ', '.join(self.prefixes))

return '\n'.join(out)

class LoreAttestor:
    def __init__(self, keyid):
        self.keyid = keyid
        self.uids = list()

def __repr__(self):
        out = list()
        out.append('  keyid: %s' % self.keyid)
        for uid in self.uids:
            out.append('    uid: %s <%s>' % uid)
        return '\n'.join(out)

class LoreAttestorDKIM(LoreAttestor):
    def __init__(self, keyid):
        self.mode = 'domain'
        super().__init__(keyid)

def get_trailer(self, fromaddr=None): # noqa
        if fromaddr:
            return 'DKIM/%s (From: %s)' % (self.keyid, fromaddr)
        return 'DKIM/%s' % self.keyid

class LoreAttestorPGP(LoreAttestor):
    def __init__(self, keyid):
        super().__init__(keyid)
        self.mode = 'person'
        self.load_subkey_uids()

def load_subkey_uids(self):
        global SUBKEY_DATA
        if self.keyid not in SUBKEY_DATA:
            gpgargs = ['--with-colons', '--list-keys', self.keyid]
            ecode, out, err = gpg_run_command(gpgargs)
            if ecode > 0:
                logger.critical('ERROR: Unable to get UIDs list matching key %s', self.keyid)
                return

keyinfo = out.decode()

uids = list()
            for line in keyinfo.split('\n'):
                if line[:4] != 'uid:':
                    continue
                chunks = line.split(':')
                if chunks[1] in ('r',):
                    # Revoked UID, ignore
                    continue
                uids.append(chunks[9])
            SUBKEY_DATA[self.keyid] = email.utils.getaddresses(uids)

self.uids = SUBKEY_DATA[self.keyid]

def get_primary_uid(self):
        return self.uids[0]

def get_matching_uid(self, fromaddr):
        for uid in self.uids:
            if fromaddr == uid[1]:
                return uid

logger.debug('No exact match, returning primary UID')
        return self.uids[0]

def get_trailer(self, fromaddr):
        if fromaddr:
            uid = self.get_matching_uid(fromaddr)
        else:
            uid = self.uids[0]

return '%s <%s> (pgp: %s)' % (uid[0], uid[1], self.keyid)

class LoreAttestationSignature:
    def __init__(self, msg):
        self.msg = msg
        self.mode = None
        self.present = False
        self.good = False
        self.valid = False
        self.trusted = False
        self.passing = False
        self.sigdate = None
        self.attestor = None
        self.errors = set()

config = get_main_config()
        try:
            driftd = int(config['attestation-staleness-days'])
        except ValueError:
            driftd = 30

self.maxdrift = datetime.timedelta(days=driftd)

def verify_time_drift(self) -> None:
        msgdt = email.utils.parsedate_to_datetime(str(self.msg['Date']))
        sdrift = self.sigdate - msgdt
        if sdrift > self.maxdrift:
            self.passing = False
            self.errors.add('Time drift between Date and t too great (%s)' % sdrift)
            return
        logger.debug('PASS : time drift between Date and t (%s)', sdrift)

def verify_identity_domain(self, identity: str, domain: str):
        # Domain is supposed to be present in identity
        if not identity.endswith(domain):
            logger.debug('domain (d=%s) is not in identity (i=%s)', domain, identity)
            self.passing = False
            return
        fromeml = email.utils.getaddresses(self.msg.get_all('from', []))[0][1]
        if identity.find('@') < 0:
            logger.debug('identity must contain @ (i=%s)', identity)
            self.passing = False
            return
        ilocal, idomain = identity.split('@')
        # identity is supposed to be present in from
        if not fromeml.endswith(f'@{idomain}'):
            self.errors.add('identity (i=%s) does not match from (from=%s)' % (identity, fromeml))
            self.passing = False
            return
        logger.debug('identity and domain match From header')

# @staticmethod
    # def get_dkim_key(domain: str, selector: str, timeout: int = 5) -> Tuple[str, str]:
    #     global DNSCACHE
    #     if (domain, selector) in DNSCACHE:
    #         return DNSCACHE[(domain, selector)]
    #
    #     name = f'{selector}._domainkey.{domain}.'
    #     logger.debug('DNS-lookup: %s', name)
    #     keydata = None
    #     try:
    #         a = dns.resolver.resolve(name, dns.rdatatype.TXT, raise_on_no_answer=False, lifetime=timeout) # noqa
    #         # Find v=DKIM1
    #         for r in a.response.answer:
    #             if r.rdtype == dns.rdatatype.TXT:
    #                 for item in r.items:
    #                     # Concatenate all strings
    #                     txtdata = b''.join(item.strings)
    #                     if txtdata.find(b'v=DKIM1') >= 0:
    #                         keydata = txtdata.decode()
    #                         break
    #             if keydata:
    #                 break
    #     except dns.resolver.NXDOMAIN: # noqa
    #         raise LookupError('Domain %s does not exist', name)
    #
    #     if not keydata:
    #         raise LookupError('Domain %s does not contain a DKIM record', name)
    #
    #     parts = get_parts_from_header(keydata)
    #     if 'p' not in parts:
    #         raise LookupError('Domain %s does not contain a DKIM key', name)
    #     if 'k' not in parts:
    #         raise LookupError('Domain %s does not indicate key time', name)
    #
    #     DNSCACHE[(domain, selector)] = (parts['k'], parts['p'])
    #     logger.debug('k=%s, p=%s', parts['k'], parts['p'])
    #     return parts['k'], parts['p']

def __repr__(self):
        out = list()
        out.append('   mode: %s' % self.mode)
        out.append('present: %s' % self.present)
        out.append('   good: %s' % self.good)
        out.append('  valid: %s' % self.valid)
        out.append('trusted: %s' % self.trusted)
        if self.attestor is not None:
            out.append('  attestor: %s' % self.attestor.keyid)

out.append('  --- validation errors ---')
        for error in self.errors:
            out.append('  | %s' % error)
        return '\n'.join(out)

class LoreAttestationSignatureDKIM(LoreAttestationSignature):
    def __init__(self, msg):
        super().__init__(msg)
        self.mode = 'dkim'
        # Doesn't quite work right, so just use dkimpy's native
        # self.native_verify()
        # return

ejected = set()
        while True:
            dks = self.msg.get('dkim-signature')
            if not dks:
                logger.debug('No DKIM-Signature headers in the message')
                return

self.present = True

ddata = get_parts_from_header(dks)
            self.attestor = LoreAttestorDKIM(ddata['d'])
            # Do we have a resolve method?
            if _resolver and hasattr(_resolver, 'resolve'):
                res = dkim.verify(self.msg.as_bytes(), dnsfunc=dkim_get_txt)
            else:
                res = dkim.verify(self.msg.as_bytes())
            if not res:
                # is list-archive or archived-at part of h=?
                hline = ddata.get('h')
                if hline:
                    hsigned = set(hline.lower().split(':'))
                    if 'list-archive' in hsigned or 'archived-at' in hsigned:
                        # Public-inbox inserts additional List-Archive and Archived-At headers,
                        # which breaks DKIM signatures if these headers are included in the hash.
                        # Eject the ones created by public-inbox and try again.
                        # XXX: This may no longer be necessary at some point if public-inbox takes care
                        #      of this scenario automatically:
                        #      https://public-inbox.org/meta/20201210202145.7agtcmrtl5jec42d@chatter.i7.local
                        logger.debug('Ejecting extra List-Archive headers and retrying')
                        changed = False
                        for header in reversed(self.msg._headers):  # noqa
                            hl = header[0].lower()
                            if hl in ('list-archive', 'archived-at') and hl not in ejected:
                                self.msg._headers.remove(header)  # noqa
                                ejected.add(hl)
                                changed = True
                                break
                        if changed:
                            # go for another round
                            continue

logger.debug('DKIM signature did NOT verify')
                logger.debug('Retrying with the next DKIM-Signature header, if any')
                at = 0
                for header in self.msg._headers:  # noqa
                    if header[0].lower() == 'dkim-signature':
                        del(self.msg._headers[at])  # noqa
                        break
                    at += 1
                continue

self.good = True

# Grab toplevel signature that we just verified
            self.valid = True
            self.trusted = True
            self.passing = True

if ddata.get('t'):
                self.sigdate = datetime.datetime.utcfromtimestamp(int(ddata['t'])).replace(tzinfo=datetime.timezone.utc)
            else:
                self.sigdate = email.utils.parsedate_to_datetime(str(self.msg['Date']))
            return

# def native_verify(self):
    #     dks = self.msg.get('dkim-signature')
    #     ddata = get_parts_from_header(dks)
    #     try:
    #         kt, kp = LoreAttestationSignature.get_dkim_key(ddata['d'], ddata['s'])
    #         if kt not in ('rsa',):  # 'ed25519'):
    #             logger.debug('DKIM key type %s not supported', kt)
    #             return
    #         pk = base64.b64decode(kp)
    #         sig = base64.b64decode(ddata['b'])
    #     except (LookupError, binascii.Error) as ex:
    #         logger.debug('Unable to look up DKIM key: %s', ex)
    #         return
    #
    #     headers = list()
    #
    #     for header in ddata['h'].split(':'):
    #         # For the POC, we assume 'relaxed/'
    #         hval = self.msg.get(header)
    #         if hval is None:
    #             # Missing headers are omitted by the DKIM RFC
    #             continue
    #         if ddata['c'].startswith('relaxed/'):
    #             hname, hval = dkim_canonicalize_header(header, str(self.msg.get(header)))
    #         else:
    #             hname = header
    #             hval = str(self.msg.get(header))
    #         headers.append(f'{hname}:{hval}')
    #
    #     # Now we add the dkim-signature header itself, without b= content
    #     if ddata['c'].startswith('relaxed/'):
    #         dname, dval = dkim_canonicalize_header('dkim-signature', dks)
    #     else:
    #         dname = 'DKIM-Signature'
    #         dval = dks
    #
    #     dval = dval.rsplit('; b=')[0] + '; b='
    #     headers.append(f'{dname}:{dval}')
    #     payload = ('\r\n'.join(headers)).encode()
    #     key = RSA.import_key(pk)
    #     hashed = SHA256.new(payload)
    #     try:
    #         # noinspection PyTypeChecker
    #         pkcs1_15.new(key).verify(hashed, sig)
    #     except (ValueError, TypeError):
    #         logger.debug('DKIM signature did not verify')
    #         self.errors.add('The DKIM signature did NOT verify!')
    #         return
    #
    #     self.good = True
    #     if not ddata.get('i'):
    #         ddata['i'] = '@' + ddata['d']
    #
    #     logger.debug('PASS : DKIM signature for d=%s, s=%s', ddata['d'], ddata['s'])
    #
    #     self.attestor = LoreAttestorDKIM(ddata['d'])
    #     self.valid = True
    #     self.trusted = True
    #     self.passing = True
    #
    #     self.verify_identity_domain(ddata['i'], ddata['d'])
    #     if ddata.get('t'):
    #         self.sigdate = datetime.datetime.utcfromtimestamp(int(ddata['t'])).replace(tzinfo=datetime.timezone.utc)
    #         self.verify_time_drift()
    #     else:
    #         self.sigdate = email.utils.parsedate_to_datetime(str(self.msg['Date']))

class LoreAttestationSignaturePGP(LoreAttestationSignature):
    def __init__(self, msg):
        super().__init__(msg)
        self.mode = 'pgp'

shdr = msg.get(HDR_PATCH_SIG)
        if not shdr:
            return

self.present = True
        sdata = get_parts_from_header(shdr)
        hhdr = msg.get(HDR_PATCH_HASHES)
        sig = base64.b64decode(sdata['b'])
        headers = list()
        hhname, hhval = dkim_canonicalize_header(HDR_PATCH_HASHES, str(hhdr))
        headers.append(f'{hhname}:{hhval}')
        # Now we add the sig header itself, without b= content
        shname, shval = dkim_canonicalize_header(HDR_PATCH_SIG, shdr)
        shval = shval.rsplit('; b=')[0] + '; b='
        headers.append(f'{shname}:{shval}')
        payload = ('\r\n'.join(headers)).encode()
        savefile = mkstemp('in-header-pgp-verify')[1]
        with open(savefile, 'wb') as fh:
            fh.write(sig)

gpgargs = list()
        config = get_main_config()
        trustmodel = config.get('attestation-trust-model', 'tofu')
        if trustmodel == 'tofu':
            gpgargs += ['--trust-model', 'tofu', '--tofu-default-policy', 'good']
        gpgargs += ['--verify', '--status-fd=1', savefile, '-']
        ecode, out, err = gpg_run_command(gpgargs, stdin=payload)
        os.unlink(savefile)
        output = out.decode()

self.good, self.valid, self.trusted, self.attestor, self.sigdate, self.errors = \
            validate_gpg_signature(output, trustmodel)

if self.good and self.valid and self.trusted:
            self.passing = True
            self.verify_time_drift()
            # XXX: Need to verify identity domain

class LoreAttestation:
    def __init__(self, _i, _m, _p):
        self.i = _i.hexdigest()
        self.m = _m.hexdigest()
        self.p = _p.hexdigest()
        self.ib = base64.b64encode(_i.digest()).decode()
        self.mb = base64.b64encode(_m.digest()).decode()
        self.pb = base64.b64encode(_p.digest()).decode()

self.lsig = None
        self.passing = False
        self.iv = False
        self.mv = False
        self.pv = False

@property
    def attid(self):
        return '%s-%s-%s' % (self.i[:8], self.m[:8], self.p[:8])

def __repr__(self):
        out = list()
        out.append('    i: %s' % self.i)
        out.append('    m: %s' % self.m)
        out.append('    p: %s' % self.p)
        out.append('    ib: %s' % self.ib)
        out.append('    mb: %s' % self.mb)
        out.append('    pb: %s' % self.pb)
        out.append('    iv: %s' % self.iv)
        out.append('    mv: %s' % self.mv)
        out.append('    pv: %s' % self.pv)
        out.append('  pass: %s' % self.passing)
        return '\n'.join(out)

def validate(self, msg):
        # Check if we have a X-Patch-Sig header. At this time, we only support two modes:
        # - GPG mode, which we check for fist
        # - Plain DKIM mode, which we check as fall-back
        # More modes may be coming in the future, depending on feedback.
        shdr = msg.get(HDR_PATCH_SIG)
        hhdr = msg.get(HDR_PATCH_HASHES)
        if hhdr is None:
            # Do we have a dkim signature header?
            if can_dkim_verify and msg.get('DKIM-Signature'):
                config = get_main_config()
                if config.get('attestation-check-dkim') == 'yes':
                    self.lsig = LoreAttestationSignatureDKIM(msg)
                    if self.lsig.passing:
                        self.passing = True
                        self.iv = True
                        self.mv = True
                        self.pv = True
                    return self.passing
            return None

if shdr is None:
            return None

sdata = get_parts_from_header(shdr)
        if sdata.get('m') == 'pgp':
            self.lsig = LoreAttestationSignaturePGP(msg)
            if self.lsig.passing:
                hdata = get_parts_from_header(hhdr)
                if hdata['i'] == self.ib:
                    self.iv = True
                if hdata['m'] == self.mb:
                    self.mv = True
                if hdata['p'] == self.pb:
                    self.pv = True

if self.iv and self.mv and self.pv:
                self.passing = True

if self.lsig is None:
            return None

return self.passing

def _run_command(cmdargs, stdin=None):
    logger.debug('Running %s' % ' '.join(cmdargs))

sp = subprocess.Popen(cmdargs,
                          stdout=subprocess.PIPE,
                          stdin=subprocess.PIPE,
                          stderr=subprocess.PIPE)

(output, error) = sp.communicate(input=stdin)

return sp.returncode, output, error

def gpg_run_command(args, stdin=None):
    config = get_main_config()
    cmdargs = [config['gpgbin'], '--batch', '--no-auto-key-retrieve', '--no-auto-check-trustdb']
    if config['attestation-gnupghome'] is not None:
        cmdargs += ['--homedir', config['attestation-gnupghome']]
    cmdargs += args

return _run_command(cmdargs, stdin=stdin)

def git_run_command(gitdir, args, stdin=None, logstderr=False):
    cmdargs = ['git', '--no-pager']
    if gitdir:
        if os.path.isdir(os.path.join(gitdir, '.git')):
            gitdir = os.path.join(gitdir, '.git')
        cmdargs += ['--git-dir', gitdir]
    cmdargs += args

ecode, out, err = _run_command(cmdargs, stdin=stdin)

out = out.decode(errors='replace')

if logstderr and len(err.strip()):
        err = err.decode(errors='replace')
        logger.debug('Stderr: %s', err)
        out += err

return ecode, out

def git_get_command_lines(gitdir, args):
    ecode, out = git_run_command(gitdir, args)
    lines = list()
    if out:
        for line in out.split('\n'):
            if line == '':
                continue
            lines.append(line)

return lines

@contextmanager
def git_temp_worktree(gitdir=None):
    """Context manager that creates a temporary work tree and chdirs into it. The
    worktree is deleted when the contex manager is closed. Taken from gj_tools."""
    dfn = None
    try:
        with TemporaryDirectory() as dfn:
            git_run_command(gitdir, ['worktree', 'add', '--detach', '--no-checkout', dfn])
            with in_directory(dfn):
                yield
    finally:
        if dfn is not None:
            git_run_command(gitdir, ['worktree', 'remove', dfn])

@contextmanager
def in_directory(dirname):
    """Context manager that chdirs into a directory and restores the original
    directory when closed. Taken from gj_tools."""
    cdir = os.getcwd()
    try:
        os.chdir(dirname)
        yield True
    finally:
        os.chdir(cdir)

def get_config_from_git(regexp, defaults=None):
    args = ['config', '-z', '--get-regexp', regexp]
    ecode, out = git_run_command(None, args)
    gitconfig = defaults
    if not gitconfig:
        gitconfig = dict()
    if not out:
        return gitconfig

for line in out.split('\x00'):
        if not line:
            continue
        key, value = line.split('\n', 1)
        try:
            chunks = key.split('.')
            cfgkey = chunks[-1]
            gitconfig[cfgkey.lower()] = value
        except ValueError:
            logger.debug('Ignoring git config entry %s', line)

return gitconfig

def get_main_config():
    global MAIN_CONFIG
    if MAIN_CONFIG is None:
        config = get_config_from_git(r'b4\..*', defaults=DEFAULT_CONFIG)
        # Legacy name was get-lore-mbox, so load those as well
        config = get_config_from_git(r'get-lore-mbox\..*', defaults=config)
        config['trailer-order'] = config['trailer-order'].split(',')
        if config['gpgbin'] is None:
            gpgcfg = get_config_from_git(r'gpg\..*', {'program': 'gpg'})
            config['gpgbin'] = gpgcfg['program']
        MAIN_CONFIG = config
    return MAIN_CONFIG

def get_data_dir():
    if 'XDG_DATA_HOME' in os.environ:
        datahome = os.environ['XDG_DATA_HOME']
    else:
        datahome = os.path.join(str(Path.home()), '.local', 'share')
    datadir = os.path.join(datahome, 'b4')
    Path(datadir).mkdir(parents=True, exist_ok=True)
    return datadir

def get_cache_dir():
    global _CACHE_CLEANED
    if 'XDG_CACHE_HOME' in os.environ:
        cachehome = os.environ['XDG_CACHE_HOME']
    else:
        cachehome = os.path.join(str(Path.home()), '.cache')
    cachedir = os.path.join(cachehome, 'b4')
    Path(cachedir).mkdir(parents=True, exist_ok=True)
    if _CACHE_CLEANED:
        return cachedir

# Delete all .mbx and .lookup files older than cache-expire
    config = get_main_config()
    try:
        expmin = int(config['cache-expire']) * 60
    except ValueError:
        logger.critical('ERROR: cache-expire must be an integer (minutes): %s', config['cache-expire'])
        expmin = 600
    expage = time.time() - expmin
    for entry in os.listdir(cachedir):
        if entry.find('.mbx') <= 0 and entry.find('.lookup') <= 0:
            continue
        st = os.stat(os.path.join(cachedir, entry))
        if st.st_mtime < expage:
            logger.debug('Cleaning up cache: %s', entry)
            os.unlink(os.path.join(cachedir, entry))
    _CACHE_CLEANED = True
    return cachedir

def get_cache_file(identifier, suffix=None):
    cachedir = get_cache_dir()
    cachefile = hashlib.sha1(identifier.encode()).hexdigest()
    if suffix:
        cachefile = f'{cachefile}.{suffix}'
    return os.path.join(cachedir, cachefile)

def get_cache(identifier, suffix=None):
    fullpath = get_cache_file(identifier, suffix=suffix)
    try:
        with open(fullpath) as fh:
            logger.debug('Using cache %s for %s', fullpath, identifier)
            return fh.read()
    except FileNotFoundError:
        logger.debug('Cache miss for %s', identifier)
    return None

def save_cache(contents, identifier, suffix=None, mode='w'):
    fullpath = get_cache_file(identifier, suffix=suffix)
    if not contents:
        # noinspection PyBroadException
        try:
            os.unlink(fullpath)
            logger.debug('Removed cache %s for %s', fullpath, identifier)
        except:
            pass
    try:
        with open(fullpath, mode) as fh:
            fh.write(contents)
            logger.debug('Saved cache %s for %s', fullpath, identifier)
    except FileNotFoundError:
        logger.debug('Could not write cache %s for %s', fullpath, identifier)

def get_user_config():
    global USER_CONFIG
    if USER_CONFIG is None:
        USER_CONFIG = get_config_from_git(r'user\..*')
        if 'name' not in USER_CONFIG:
            udata = pwd.getpwuid(os.getuid())
            USER_CONFIG['name'] = udata.pw_gecos
    return USER_CONFIG

def get_requests_session():
    global REQSESSION
    if REQSESSION is None:
        REQSESSION = requests.session()
        REQSESSION.headers.update({'User-Agent': 'b4/%s' % __VERSION__})
    return REQSESSION

def get_msgid_from_stdin():
    if not sys.stdin.isatty():
        message = email.message_from_string(sys.stdin.read())
        return message.get('Message-ID', None)
    logger.error('Error: pipe a message or pass msgid as parameter')
    sys.exit(1)

def get_msgid(cmdargs):
    if not cmdargs.msgid:
        logger.debug('Getting Message-ID from stdin')
        msgid = get_msgid_from_stdin()
        if msgid is None:
            logger.error('Unable to find a valid message-id in stdin.')
            sys.exit(1)
    else:
        msgid = cmdargs.msgid

msgid = msgid.strip('<>')
    # Handle the case when someone pastes a full URL to the message
    matches = re.search(r'^https?://[^/]+/([^/]+)/([^/]+@[^/]+)', msgid, re.IGNORECASE)
    if matches:
        chunks = matches.groups()
        msgid = urllib.parse.unquote(chunks[1])
        # Infer the project name from the URL, if possible
        if chunks[0] != 'r':
            cmdargs.useproject = chunks[0]
    # Handle special case when msgid is prepended by id: or rfc822msgid:
    if msgid.find('id:') >= 0:
        msgid = re.sub(r'^\w*id:', '', msgid)

return msgid

def save_strict_thread(in_mbx, out_mbx, msgid):
    want = {msgid}
    got = set()
    seen = set()
    maybe = dict()
    while True:
        for msg in in_mbx:
            c_msgid = LoreMessage.get_clean_msgid(msg)
            seen.add(c_msgid)
            if c_msgid in got:
                continue
            logger.debug('Looking at: %s', c_msgid)

refs = set()
            msgrefs = list()
            if msg.get('In-Reply-To', None):
                msgrefs += email.utils.getaddresses([str(x) for x in msg.get_all('in-reply-to', [])])
            if msg.get('References', None):
                msgrefs += email.utils.getaddresses([str(x) for x in msg.get_all('references', [])])
            for ref in set([x[1] for x in msgrefs]):
                if ref in got or ref in want:
                    want.add(c_msgid)
                elif len(ref):
                    refs.add(ref)
                    if c_msgid not in want:
                        if ref not in maybe:
                            maybe[ref] = set()
                        logger.debug('Going into maybe: %s->%s', ref, c_msgid)
                        maybe[ref].add(c_msgid)

if c_msgid in want:
                out_mbx.add(msg)
                got.add(c_msgid)
                want.update(refs)
                want.discard(c_msgid)
                logger.debug('Kept in thread: %s', c_msgid)
                if c_msgid in maybe:
                    # Add all these to want
                    want.update(maybe[c_msgid])
                    maybe.pop(c_msgid)
                # Add all maybes that have the same ref into want
                for ref in refs:
                    if ref in maybe:
                        want.update(maybe[ref])
                        maybe.pop(ref)

# Remove any entries not in "seen" (missing messages)
        for c_msgid in set(want):
            if c_msgid not in seen or c_msgid in got:
                want.remove(c_msgid)
        if not len(want):
            break

if not len(out_mbx):
        return None

if len(in_mbx) > len(out_mbx):
        logger.debug('Reduced mbox to strict matches only (%s->%s)', len(in_mbx), len(out_mbx))

def get_pi_thread_by_url(t_mbx_url, savefile, nocache=False):
    cachefile = get_cache_file(t_mbx_url, 'pi.mbx')
    if os.path.exists(cachefile) and not nocache:
        logger.debug('Using cached copy: %s', cachefile)
        shutil.copyfile(cachefile, savefile)
        return savefile
    session = get_requests_session()
    resp = session.get(t_mbx_url)
    if resp.status_code != 200:
        logger.critical('Server returned an error: %s', resp.status_code)
        return None
    t_mbox = gzip.decompress(resp.content)
    resp.close()
    if not len(t_mbox):
        logger.critical('No messages found for that query')
        return None
    # Convert mboxrd to mboxo that python understands
    t_mbox = t_mbox.replace(b'\n>>From ', b'\n>From ')
    with open(savefile, 'wb') as fh:
        logger.debug('Saving %s', savefile)
        fh.write(t_mbox)
    shutil.copyfile(savefile, cachefile)
    return savefile

def get_pi_thread_by_msgid(msgid, savefile, useproject=None, nocache=False):
    qmsgid = urllib.parse.quote_plus(msgid)
    config = get_main_config()
    # Grab the head from lore, to see where we are redirected
    midmask = config['midmask'] % qmsgid
    loc = urllib.parse.urlparse(midmask)
    if useproject:
        projurl = '%s://%s/%s' % (loc.scheme, loc.netloc, useproject)
    else:
        logger.info('Looking up %s', midmask)
        session = get_requests_session()
        resp = session.head(midmask)
        if resp.status_code < 300 or resp.status_code > 400:
            logger.critical('That message-id is not known.')
            return None
        # Pop msgid from the end of the redirect
        chunks = resp.headers['Location'].rstrip('/').split('/')
        projurl = '/'.join(chunks[:-1])
        resp.close()
    t_mbx_url = '%s/%s/t.mbox.gz' % (projurl, qmsgid)
    logger.debug('t_mbx_url=%s', t_mbx_url)

logger.critical('Grabbing thread from %s', projurl.split('://')[1])

tmp_mbox = mkstemp('b4-lookup-mbox')[1]
    in_mbxf = get_pi_thread_by_url(t_mbx_url, tmp_mbox, nocache=nocache)
    if not in_mbxf:
        os.unlink(tmp_mbox)
        return None
    in_mbx = mailbox.mbox(in_mbxf)
    out_mbx = mailbox.mbox(savefile)
    save_strict_thread(in_mbx, out_mbx, msgid)
    in_mbx.close()
    out_mbx.close()
    os.unlink(in_mbxf)
    return savefile

def git_format_patches(gitdir, start, end, reroll=None):
    gitargs = ['format-patch', '--stdout']
    if reroll is not None:
        gitargs += ['-v', str(reroll)]
    gitargs += ['%s..%s' % (start, end)]
    ecode, out = git_run_command(gitdir, gitargs)
    return ecode, out

def git_commit_exists(gitdir, commit_id):
    gitargs = ['cat-file', '-e', commit_id]
    ecode, out = git_run_command(gitdir, gitargs)
    return ecode == 0

def git_branch_contains(gitdir, commit_id):
    gitargs = ['branch', '--format=%(refname:short)', '--contains', commit_id]
    lines = git_get_command_lines(gitdir, gitargs)
    return lines

def format_addrs(pairs, clean=True):
    addrs = set()
    for pair in pairs:
        pair = list(pair)
        if pair[0] == pair[1]:
            pair[0] = ''
        if clean:
            # Remove any quoted-printable header junk from the name
            pair[0] = LoreMessage.clean_header(pair[0])
        addrs.add(email.utils.formataddr(pair))  # noqa
    return ', '.join(addrs)

def make_quote(body, maxlines=5):
    headers, message, trailers, basement, signature = LoreMessage.get_body_parts(body)
    if not len(message):
        # Sometimes there is no message, just trailers
        return '> \n'
    # Remove common greetings
    message = re.sub(r'^(hi|hello|greetings|dear)\W.*\n+', '', message, flags=re.I)
    quotelines = list()
    qcount = 0
    for line in message.split('\n'):
        # Quote the first paragraph only and then [snip] if we quoted more than maxlines
        if qcount > maxlines and not len(line.strip()):
            quotelines.append('> ')
            quotelines.append('> [...]')
            break
        quotelines.append('> %s' % line.rstrip())
        qcount += 1
    return '\n'.join(quotelines)

def parse_int_range(intrange, upper=None):
    # Remove all whitespace
    intrange = re.sub(r'\s', '', intrange)
    for n in intrange.split(','):
        if n.isdigit():
            yield int(n)
        elif n.find('<') == 0 and len(n) > 1 and n[1:].isdigit():
            yield from range(1, int(n[1:]))
        elif n.find('-') > 0:
            nr = n.split('-')
            if nr[0].isdigit() and nr[1].isdigit():
                yield from range(int(nr[0]), int(nr[1])+1)
            elif not len(nr[1]) and nr[0].isdigit() and upper:
                yield from range(int(nr[0]), upper+1)
        else:
            logger.critical('Unknown range value specified: %s', n)

def dkim_canonicalize_header(hname, hval):
    hname = hname.lower()
    hval = hval.strip()
    hval = re.sub(r'\n', '', hval)
    hval = re.sub(r'\s+', ' ', hval)
    return hname, hval

def get_parts_from_header(hstr: str) -> dict:
    hstr = re.sub(r'\s*', '', hstr)
    hdata = dict()
    for chunk in hstr.split(';'):
        parts = chunk.split('=', 1)
        if len(parts) < 2:
            continue
        hdata[parts[0]] = parts[1]
    return hdata

def validate_gpg_signature(output, trustmodel):
    good = False
    valid = False
    trusted = False
    attestor = None
    sigdate = None
    errors = set()
    gs_matches = re.search(r'^\[GNUPG:] GOODSIG ([0-9A-F]+)\s+.*$', output, re.M)
    if gs_matches:
        logger.debug('  GOODSIG')
        good = True
        keyid = gs_matches.groups()[0]
        attestor = LoreAttestorPGP(keyid)
        puid = '%s <%s>' % attestor.get_primary_uid()
        vs_matches = re.search(r'^\[GNUPG:] VALIDSIG ([0-9A-F]+) (\d{4}-\d{2}-\d{2}) (\d+)', output, re.M)
        if vs_matches:
            logger.debug('  VALIDSIG')
            valid = True
            ymd = vs_matches.groups()[1]
            sigdate = datetime.datetime.strptime(ymd, '%Y-%m-%d').replace(tzinfo=datetime.timezone.utc)
            # Do we have a TRUST_(FULLY|ULTIMATE)?
            ts_matches = re.search(r'^\[GNUPG:] TRUST_(FULLY|ULTIMATE)', output, re.M)
            if ts_matches:
                logger.debug('  TRUST_%s', ts_matches.groups()[0])
                trusted = True
            else:
                errors.add('Insufficient trust (model=%s): %s (%s)' % (trustmodel, keyid, puid))
        else:
            errors.add('Signature not valid from key: %s (%s)' % (attestor.keyid, puid))
    else:
        # Are we missing a key?
        matches = re.search(r'^\[GNUPG:] NO_PUBKEY ([0-9A-F]+)$', output, re.M)
        if matches:
            errors.add('Missing public key: %s' % matches.groups()[0])
        # Is the key expired?
        matches = re.search(r'^\[GNUPG:] EXPKEYSIG (.*)$', output, re.M)
        if matches:
            errors.add('Expired key: %s' % matches.groups()[0])

return good, valid, trusted, attestor, sigdate, errors

def dkim_get_txt(name: bytes, timeout: int = 5):
    global _DKIM_DNS_CACHE
    if name not in _DKIM_DNS_CACHE:
        lookup = name.decode()
        logger.debug('DNS-lookup: %s', lookup)
        try:
            a = _resolver.resolve(lookup, dns.rdatatype.TXT, raise_on_no_answer=False, lifetime=timeout, search=True)
            for r in a.response.answer:
                if r.rdtype == dns.rdatatype.TXT:
                    for item in r.items:
                        # Concatenate all strings
                        txtdata = b''.join(item.strings)
                        if txtdata.find(b'p=') >= 0:
                            _DKIM_DNS_CACHE[name] = txtdata
                            return txtdata
        except dns.resolver.NXDOMAIN:
            pass
        _DKIM_DNS_CACHE[name] = None
    return _DKIM_DNS_CACHE[name]
07070100000006000081A40000273B00000064000000015FEA2051000014DF000000000000000000000000000000000000001800000000b4-0.6.2+4/b4/attest.py#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright (C) 2020 by the Linux Foundation
#

import sys
import email
import email.utils
import email.message
import email.header
import b4
import argparse
import base64

logger = b4.logger

def in_header_attest(lmsg: b4.LoreMessage, mode: str = 'pgp', replace: bool = False) -> None:
    if lmsg.msg.get(b4.HDR_PATCH_HASHES):
        if not replace:
            logger.info(' attest: message already attested')
            return
        del lmsg.msg[b4.HDR_PATCH_HASHES]
        del lmsg.msg[b4.HDR_PATCH_SIG]

logger.info(' attest: generating attestation hashes')
    if not lmsg.attestation:
        raise RuntimeError('Could not calculate patch attestation')

headers = list()
    hparts = [
        'v=1',
        'h=sha256',
        f'i={lmsg.attestation.ib}',
        f'm={lmsg.attestation.mb}',
        f'p={lmsg.attestation.pb}',
    ]
    if lmsg.git_patch_id:
        hparts.append(f'g={lmsg.git_patch_id}')

hhname, hhval = b4.dkim_canonicalize_header(b4.HDR_PATCH_HASHES, '; '.join(hparts))
    headers.append(f'{hhname}:{hhval}')

logger.debug('Signing with mode=%s', mode)
    if mode == 'pgp':
        usercfg = b4.get_user_config()
        keyid = usercfg.get('signingkey')
        identity = usercfg.get('email')
        if not identity:
            raise RuntimeError('Please set user.email to use this feature')
        if not keyid:
            raise RuntimeError('Please set user.signingKey to use this feature')

logger.debug('Using i=%s, s=0x%s', identity, keyid.rstrip('!'))
        gpgargs = ['-b', '-u', f'{keyid}']

hparts = [
            'm=pgp',
            f'i={identity}',
            's=0x%s' % keyid.rstrip('!'),
            'b=',
        ]

shname, shval = b4.dkim_canonicalize_header(b4.HDR_PATCH_SIG, '; '.join(hparts))
        headers.append(f'{shname}:{shval}')
        payload = '\r\n'.join(headers).encode()
        ecode, out, err = b4.gpg_run_command(gpgargs, payload)
        if ecode > 0:
            logger.critical('Running gpg failed')
            logger.critical(err.decode())
            raise RuntimeError('Running gpg failed')
        bdata = base64.b64encode(out).decode()
        shval += header_splitter(bdata)
    else:
        raise NotImplementedError('Mode %s not implemented' % mode)

hhdr = email.header.make_header([(hhval.encode(), 'us-ascii')], maxlinelen=78)
    shdr = email.header.make_header([(shval.encode(), 'us-ascii')], maxlinelen=78)
    lmsg.msg[b4.HDR_PATCH_HASHES] = hhdr
    lmsg.msg[b4.HDR_PATCH_SIG] = shdr

def header_splitter(longstr: str, limit: int = 77) -> str:
    splitstr = list()
    first = True
    while len(longstr) > limit:
        at = limit
        if first:
            first = False
            at -= 2
        splitstr.append(longstr[:at])
        longstr = longstr[at:]
    splitstr.append(longstr)
    return ' '.join(splitstr)

def attest_patches(cmdargs: argparse.Namespace) -> None:
    for pf in cmdargs.patchfile:
        with open(pf, 'rb') as fh:
            msg = email.message_from_bytes(fh.read())
        lmsg = b4.LoreMessage(msg)
        lmsg.load_hashes()
        if not lmsg.attestation:
            logger.debug('Nothing to attest in %s, skipped')
            continue
        logger.info('Attesting: %s', pf)
        in_header_attest(lmsg, replace=True)
        with open(pf, 'wb') as fh:
            fh.write(lmsg.msg.as_bytes())

def mutt_filter() -> None:
    if sys.stdin.isatty():
        logger.error('Error: Mutt mode expects a message on stdin')
        sys.exit(1)
    inb = sys.stdin.buffer.read()
    # Quick exit if we don't find x-patch-sig
    if inb.find(b'X-Patch-Sig:') < 0:
        sys.stdout.buffer.write(inb)
        return
    msg = email.message_from_bytes(inb)
    try:
        if msg.get('x-patch-sig'):
            lmsg = b4.LoreMessage(msg)
            lmsg.load_hashes()
            latt = lmsg.attestation
            if latt:
                if latt.validate(msg):
                    trailer = latt.lsig.attestor.get_trailer(lmsg.fromemail)
                    msg.add_header('Attested-By', trailer)
                elif latt.lsig:
                    if not latt.lsig.errors:
                        failed = list()
                        if not latt.pv:
                            failed.append('patch content')
                        if not latt.mv:
                            failed.append('commit message')
                        if not latt.iv:
                            failed.append('patch metadata')
                        latt.lsig.errors.add('signature failed (%s)' % ', '.join(failed))
                    msg.add_header('Attestation-Failed', ', '.join(latt.lsig.errors))
            # Delete the x-patch-hashes and x-patch-sig headers so
            # they don't boggle up the view
            for i in reversed(range(len(msg._headers))):  # noqa
                hdrName = msg._headers[i][0].lower()  # noqa
                if hdrName in ('x-patch-hashes', 'x-patch-sig'):
                    del msg._headers[i]  # noqa
    except:  # noqa
        # Don't prevent email from being displayed even if we died horribly
        sys.stdout.buffer.write(inb)
        return

sys.stdout.buffer.write(msg.as_bytes(policy=b4.emlpolicy))
07070100000007000081A40000273B00000064000000015FEA205100002D18000000000000000000000000000000000000001900000000b4-0.6.2+4/b4/command.py#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright (C) 2020 by the Linux Foundation
#
__author__ = 'Konstantin Ryabitsev <konstantin@linuxfoundation.org>'

import argparse
import logging
import b4
import sys

logger = b4.logger

def cmd_mbox_common_opts(sp):
    sp.add_argument('msgid', nargs='?',
                    help='Message ID to process, or pipe a raw message')
    sp.add_argument('-o', '--outdir', default='.',
                    help='Output into this directory (or use - to output mailbox contents to stdout)')
    sp.add_argument('-p', '--use-project', dest='useproject', default=None,
                    help='Use a specific project instead of guessing (linux-mm, linux-hardening, etc)')
    sp.add_argument('-c', '--check-newer-revisions', dest='checknewer', action='store_true', default=False,
                    help='Check if newer patch revisions exist')
    sp.add_argument('-n', '--mbox-name', dest='wantname', default=None,
                    help='Filename to name the mbox file')
    sp.add_argument('-m', '--use-local-mbox', dest='localmbox', default=None,
                    help='Instead of grabbing a thread from lore, process this mbox file')
    sp.add_argument('-C', '--no-cache', dest='nocache', action='store_true', default=False,
                    help='Do not use local cache')

def cmd_mbox(cmdargs):
    import b4.mbox
    b4.mbox.main(cmdargs)

def cmd_am(cmdargs):
    import b4.mbox
    b4.mbox.main(cmdargs)

def cmd_attest(cmdargs):
    import b4.attest
    if cmdargs.mutt_filter:
        b4.attest.mutt_filter()
    elif len(cmdargs.patchfile):
        b4.attest.attest_patches(cmdargs)
    else:
        logger.critical('ERROR: missing patches to attest')
        sys.exit(1)

def cmd_pr(cmdargs):
    import b4.pr
    b4.pr.main(cmdargs)

def cmd_ty(cmdargs):
    import b4.ty
    b4.ty.main(cmdargs)

def cmd_diff(cmdargs):
    import b4.diff
    b4.diff.main(cmdargs)

def cmd():
    # noinspection PyTypeChecker
    parser = argparse.ArgumentParser(
        prog='b4',
        description='A tool to work with public-inbox patches',
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
    parser.add_argument('--version', action='version', version=b4.__VERSION__)
    parser.add_argument('-d', '--debug', action='store_true', default=False,
                        help='Add more debugging info to the output')
    parser.add_argument('-q', '--quiet', action='store_true', default=False,
                        help='Output critical information only')

subparsers = parser.add_subparsers(help='sub-command help', dest='subcmd')

# b4 mbox
    sp_mbox = subparsers.add_parser('mbox', help='Download a thread as an mbox file')
    cmd_mbox_common_opts(sp_mbox)
    sp_mbox.add_argument('-f', '--filter-dupes', dest='filterdupes', action='store_true', default=False,
                         help='When adding messages to existing maildir, filter out duplicates')
    sp_mbox.set_defaults(func=cmd_mbox)

# b4 am
    sp_am = subparsers.add_parser('am', help='Create an mbox file that is ready to git-am')
    cmd_mbox_common_opts(sp_am)
    sp_am.add_argument('-v', '--use-version', dest='wantver', type=int, default=None,
                       help='Get a specific version of the patch/series')
    sp_am.add_argument('-t', '--apply-cover-trailers', dest='covertrailers', action='store_true', default=False,
                       help='Apply trailers sent to the cover letter to all patches')
    sp_am.add_argument('-S', '--sloppy-trailers', dest='sloppytrailers', action='store_true', default=False,
                       help='Apply trailers without email address match checking')
    sp_am.add_argument('-T', '--no-add-trailers', dest='noaddtrailers', action='store_true', default=False,
                       help='Do not add or sort any trailers')
    sp_am.add_argument('-s', '--add-my-sob', dest='addmysob', action='store_true', default=False,
                       help='Add your own signed-off-by to every patch')
    sp_am.add_argument('-l', '--add-link', dest='addlink', action='store_true', default=False,
                       help='Add a lore.kernel.org/r/ link to every patch')
    sp_am.add_argument('-Q', '--quilt-ready', dest='quiltready', action='store_true', default=False,
                       help='Save mbox patches in a quilt-ready folder')
    sp_am.add_argument('-P', '--cherry-pick', dest='cherrypick', default=None,
                       help='Cherry-pick a subset of patches (e.g. "-P 1-2,4,6-", '
                            '"-P _" to use just the msgid specified, or '
                            '"-P *globbing*" to match on commit subject)')
    sp_am.add_argument('-g', '--guess-base', dest='guessbase', action='store_true', default=False,
                       help='Try to guess the base of the series (if not specified)')
    sp_am.add_argument('-3', '--prep-3way', dest='threeway', action='store_true', default=False,
                       help='Prepare for a 3-way merge '
                            '(tries to ensure that all index blobs exist by making a fake commit range)')
    sp_am.add_argument('--cc-trailers', dest='copyccs', action='store_true', default=False,
                       help='Copy all Cc\'d addresses into Cc: trailers')
    sp_am.add_argument('--no-cover', dest='nocover', action='store_true', default=False,
                       help='Do not save the cover letter (on by default when using -o -)')
    sp_am.set_defaults(func=cmd_am)

# b4 attest
    sp_att = subparsers.add_parser('attest', help='Create cryptographic attestation for a set of patches')
    sp_att.add_argument('-f', '--from', dest='sender', default=None,
                        help='OBSOLETE: this option does nothing and will be removed')
    sp_att.add_argument('-n', '--no-submit', dest='nosubmit', action='store_true', default=False,
                        help='OBSOLETE: this option does nothing and will be removed')
    sp_att.add_argument('-o', '--output', default=None,
                        help='OBSOLETE: this option does nothing and will be removed')
    sp_att.add_argument('-m', '--mutt-filter', action='store_true', default=False,
                        help='Run in mutt filter mode')
    sp_att.add_argument('patchfile', nargs='*', help='Patches to attest')
    sp_att.set_defaults(func=cmd_attest)

# b4 pr
    sp_pr = subparsers.add_parser('pr', help='Fetch a pull request found in a message ID')
    sp_pr.add_argument('-g', '--gitdir', default=None,
                       help='Operate on this git tree instead of current dir')
    sp_pr.add_argument('-b', '--branch', default=None,
                       help='Check out FETCH_HEAD into this branch after fetching')
    sp_pr.add_argument('-c', '--check', action='store_true', default=False,
                       help='Check if pull request has already been applied')
    sp_pr.add_argument('-e', '--explode', action='store_true', default=False,
                       help='Convert a pull request into an mbox full of patches')
    sp_pr.add_argument('-o', '--output-mbox', dest='outmbox', default=None,
                       help='Save exploded messages into this mailbox (default: msgid.mbx)')
    sp_pr.add_argument('msgid', nargs='?',
                       help='Message ID to process, or pipe a raw message')
    sp_pr.set_defaults(func=cmd_pr)

# b4 ty
    sp_ty = subparsers.add_parser('ty', help='Generate thanks email when something gets merged/applied')
    sp_ty.add_argument('-g', '--gitdir', default=None,
                       help='Operate on this git tree instead of current dir')
    sp_ty.add_argument('-o', '--outdir', default='.',
                       help='Write thanks files into this dir (default=.)')
    sp_ty.add_argument('-l', '--list', action='store_true', default=False,
                       help='List pull requests and patch series you have retrieved')
    sp_ty.add_argument('-s', '--send', default=None,
                       help='Generate thankyous for specific entries from -l (e.g.: 1,3-5,7-; or "all")')
    sp_ty.add_argument('-d', '--discard', default=None,
                       help='Discard specific messages from -l (e.g.: 1,3-5,7-; or "all")')
    sp_ty.add_argument('-a', '--auto', action='store_true', default=False,
                       help='Use the Auto-Thankanator to figure out what got applied/merged')
    sp_ty.add_argument('-b', '--branch', default=None,
                       help='The branch to check against, instead of current')
    sp_ty.add_argument('--since', default='1.week',
                       help='The --since option to use when auto-matching patches (default=1.week)')
    sp_ty.set_defaults(func=cmd_ty)

# b4 diff
    sp_diff = subparsers.add_parser('diff', help='Show a range-diff to previous series revision')
    sp_diff.add_argument('msgid', nargs='?',
                         help='Message ID to process, or pipe a raw message')
    sp_diff.add_argument('-g', '--gitdir', default=None,
                         help='Operate on this git tree instead of current dir')
    sp_diff.add_argument('-p', '--use-project', dest='useproject', default=None,
                         help='Use a specific project instead of guessing (linux-mm, linux-hardening, etc)')
    sp_diff.add_argument('-C', '--no-cache', dest='nocache', action='store_true', default=False,
                         help='Do not use local cache')
    sp_diff.add_argument('-v', '--compare-versions', dest='wantvers', type=int, default=None, nargs='+',
                         help='Compare specific versions instead of latest and one before that, e.g. -v 3 5')
    sp_diff.add_argument('-n', '--no-diff', dest='nodiff', action='store_true', default=False,
                         help='Do not generate a diff, just show the command to do it')
    sp_diff.add_argument('-o', '--output-diff', dest='outdiff', default=None,
                         help='Save diff into this file instead of outputting to stdout')
    sp_diff.add_argument('-c', '--color', dest='color', action='store_true', default=False,
                         help='Force color output even when writing to file')
    sp_diff.add_argument('-m', '--compare-am-mboxes', dest='ambox', nargs=2, default=None,
                         help='Compare two mbx files prepared with "b4 am"')
    sp_diff.set_defaults(func=cmd_diff)

cmdargs = parser.parse_args()

logger.setLevel(logging.DEBUG)

ch = logging.StreamHandler()
    formatter = logging.Formatter('%(message)s')
    ch.setFormatter(formatter)

if cmdargs.quiet:
        ch.setLevel(logging.CRITICAL)
    elif cmdargs.debug:
        ch.setLevel(logging.DEBUG)
    else:
        ch.setLevel(logging.INFO)

logger.addHandler(ch)

if 'func' not in cmdargs:
        parser.print_help()
        sys.exit(1)

cmdargs.func(cmdargs)

if __name__ == '__main__':
    # We're running from a checkout, so reflect git commit in the version
    import os
    # noinspection PyBroadException
    try:
        if b4.__VERSION__.find('-dev') > 0:
            base = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
            dotgit = os.path.join(base, '.git')
            ecode, short = b4.git_run_command(dotgit, ['rev-parse', '--short', 'HEAD'])
            if ecode == 0:
                b4.__VERSION__ = '%s-%.5s' % (b4.__VERSION__, short.strip())
    except Exception as ex:
        # Any failures above are non-fatal
        pass
    cmd()
07070100000008000081A40000273B00000064000000015FEA2051000013E4000000000000000000000000000000000000001600000000b4-0.6.2+4/b4/diff.py#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright (C) 2020 by the Linux Foundation
#
__author__ = 'Konstantin Ryabitsev <konstantin@linuxfoundation.org>'

import os
import sys
import b4
import b4.mbox
import mailbox
import shutil

from tempfile import mkstemp

logger = b4.logger

def diff_same_thread_series(cmdargs):
    msgid = b4.get_msgid(cmdargs)
    wantvers = cmdargs.wantvers
    if wantvers and len(wantvers) > 2:
        logger.critical('Can only compare two versions at a time')
        sys.exit(1)

# start by grabbing the mbox provided
    savefile = mkstemp('b4-diff-to')[1]
    # Do we have a cache of this lookup?
    identifier = msgid
    if wantvers:
        identifier += '-' + '-'.join([str(x) for x in wantvers])
    if cmdargs.useproject:
        identifier += '-' + cmdargs.useproject

cachefile = b4.get_cache_file(identifier, suffix='diff.mbx')
    if os.path.exists(cachefile) and not cmdargs.nocache:
        logger.info('Using cached copy of the lookup')
        shutil.copyfile(cachefile, savefile)
        mboxfile = savefile
    else:
        mboxfile = b4.get_pi_thread_by_msgid(msgid, savefile, useproject=cmdargs.useproject, nocache=cmdargs.nocache)
        if mboxfile is None:
            logger.critical('Unable to retrieve thread: %s', msgid)
            return
        b4.mbox.get_extra_series(mboxfile, direction=-1, wantvers=wantvers)
        shutil.copyfile(mboxfile, cachefile)

mbx = mailbox.mbox(mboxfile)
    count = len(mbx)
    logger.info('---')
    logger.info('Analyzing %s messages in the thread', count)
    lmbx = b4.LoreMailbox()
    for key, msg in mbx.items():
        lmbx.add_message(msg)

mbx.close()
    os.unlink(savefile)

if wantvers and len(wantvers) == 1:
        upper = max(lmbx.series.keys())
        lower = wantvers[0]
    elif wantvers and len(wantvers) == 2:
        upper = max(wantvers)
        lower = min(wantvers)
    else:
        upper = max(lmbx.series.keys())
        lower = min(lmbx.series.keys())

if upper == lower:
        logger.critical('ERROR: Could not auto-find previous revision')
        logger.critical('       Run "b4 am -T" manually, then "b4 diff -m mbx1 mbx2"')
        return None, None

if upper not in lmbx.series:
        return None, None

if lower not in lmbx.series:
        return None, None

if not lmbx.series[lower].complete:
        lmbx.backfill(lower)

if not lmbx.series[upper].complete:
        lmbx.backfill(upper)

return lmbx.series[lower], lmbx.series[upper]

def diff_mboxes(cmdargs):
    chunks = list()
    for mboxfile in cmdargs.ambox:
        if not os.path.exists(mboxfile):
            logger.critical('Cannot open %s', mboxfile)
            return None, None

mbx = mailbox.mbox(mboxfile)
        count = len(mbx)
        logger.info('Loading %s messages from %s', count, mboxfile)
        lmbx = b4.LoreMailbox()
        for key, msg in mbx.items():
            lmbx.add_message(msg)
        if len(lmbx.series) < 1:
            logger.critical('No valid patches found in %s', mboxfile)
            sys.exit(1)
        if len(lmbx.series) > 1:
            logger.critical('More than one series version in %s, will use latest', mboxfile)

chunks.append(lmbx.series[max(lmbx.series.keys())])

return chunks

def main(cmdargs):
    if cmdargs.ambox is not None:
        lser, user = diff_mboxes(cmdargs)
    else:
        lser, user = diff_same_thread_series(cmdargs)

if lser is None or user is None:
        sys.exit(1)

# Prepare the lower fake-am range
    lsc, lec = lser.make_fake_am_range(gitdir=cmdargs.gitdir)
    if lsc is None or lec is None:
        logger.critical('---')
        logger.critical('Could not create fake-am range for lower series v%s', lser.revision)
        sys.exit(1)
    # Prepare the upper fake-am range
    usc, uec = user.make_fake_am_range(gitdir=cmdargs.gitdir)
    if usc is None or uec is None:
        logger.critical('---')
        logger.critical('Could not create fake-am range for upper series v%s', user.revision)
        sys.exit(1)
    grdcmd = 'git range-diff %.12s..%.12s %.12s..%.12s' % (lsc, lec, usc, uec)
    if cmdargs.nodiff:
        logger.info('Success, to compare v%s and v%s:', lser.revision, user.revision)
        logger.info(f'    {grdcmd}')
        sys.exit(0)
    logger.info('Diffing v%s and v%s', lser.revision, user.revision)
    logger.info('    Running: %s', grdcmd)
    gitargs = ['range-diff', f'{lsc}..{lec}', f'{usc}..{uec}']
    if cmdargs.outdiff is None or cmdargs.color:
        gitargs.append('--color')
    ecode, rdiff = b4.git_run_command(cmdargs.gitdir, gitargs)
    if ecode > 0:
        logger.critical('Unable to generate diff')
        logger.critical('Try running it yourself:')
        logger.critical(f'    {grdcmd}')
        sys.exit(1)
    if cmdargs.outdiff is not None:
        logger.info('Writing %s', cmdargs.outdiff)
        fh = open(cmdargs.outdiff, 'w')
    else:
        logger.info('---')
        fh = sys.stdout
    fh.write(rdiff)
07070100000009000081A40000273B00000064000000015FEA205100005799000000000000000000000000000000000000001600000000b4-0.6.2+4/b4/mbox.py#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright (C) 2020 by the Linux Foundation
#
__author__ = 'Konstantin Ryabitsev <konstantin@linuxfoundation.org>'

import os
import sys
import mailbox
import email
import email.message
import email.utils
import re
import time
import json
import fnmatch
import shutil

import urllib.parse
import xml.etree.ElementTree

import b4

from tempfile import mkstemp

logger = b4.logger

def mbox_to_am(mboxfile, cmdargs):
    config = b4.get_main_config()
    outdir = cmdargs.outdir
    if outdir == '-':
        cmdargs.nocover = True
    wantver = cmdargs.wantver
    wantname = cmdargs.wantname
    covertrailers = cmdargs.covertrailers
    if os.path.isdir(mboxfile):
        mbx = mailbox.Maildir(mboxfile)
    else:
        mbx = mailbox.mbox(mboxfile)
    count = len(mbx)
    logger.info('Analyzing %s messages in the thread', count)
    lmbx = b4.LoreMailbox()
    # Go through the mbox once to populate base series
    for key, msg in mbx.items():
        lmbx.add_message(msg)

lser = lmbx.get_series(revision=wantver, sloppytrailers=cmdargs.sloppytrailers)
    if lser is None and wantver is None:
        logger.critical('No patches found.')
        return
    if lser is None:
        logger.critical('Unable to find revision %s', wantver)
        return
    if len(lmbx.series) > 1 and not wantver:
        logger.info('Will use the latest revision: v%s', lser.revision)
        logger.info('You can pick other revisions using the -vN flag')

if wantname:
        slug = wantname
        if wantname.find('.') > -1:
            slug = '.'.join(wantname.split('.')[:-1])
        gitbranch = slug
    else:
        slug = lser.get_slug(extended=True)
        gitbranch = lser.get_slug(extended=False)

if outdir != '-':
        am_filename = os.path.join(outdir, '%s.mbx' % slug)
        am_cover = os.path.join(outdir, '%s.cover' % slug)

if os.path.exists(am_filename):
            os.unlink(am_filename)
    else:
        # Create a temporary file that we will remove later
        am_filename = mkstemp('b4-am-stdout')[1]
        am_cover = None

logger.info('---')
    if cmdargs.cherrypick:
        cherrypick = list()
        if cmdargs.cherrypick == '_':
            msgid = b4.get_msgid(cmdargs)
            # Only grab the exact msgid provided
            at = 0
            for lmsg in lser.patches[1:]:
                at += 1
                if lmsg and lmsg.msgid == msgid:
                    cherrypick = [at]
                    cmdargs.cherrypick = f'<{msgid}>'
                    break
            if not len(cherrypick):
                logger.critical('Specified msgid is not present in the series, cannot cherrypick')
                sys.exit(1)
        elif cmdargs.cherrypick.find('*') >= 0:
            # Globbing on subject
            at = 0
            for lmsg in lser.patches[1:]:
                at += 1
                if fnmatch.fnmatch(lmsg.subject, cmdargs.cherrypick):
                    cherrypick.append(at)
            if not len(cherrypick):
                logger.critical('Could not match "%s" to any subjects in the series', cmdargs.cherrypick)
                sys.exit(1)
        else:
            cherrypick = list(b4.parse_int_range(cmdargs.cherrypick, upper=len(lser.patches)-1))
    else:
        cherrypick = None

logger.critical('Writing %s', am_filename)
    mbx = mailbox.mbox(am_filename)
    try:
        am_mbx = lser.save_am_mbox(mbx, noaddtrailers=cmdargs.noaddtrailers,
                                   covertrailers=covertrailers, trailer_order=config['trailer-order'],
                                   addmysob=cmdargs.addmysob, addlink=cmdargs.addlink,
                                   linkmask=config['linkmask'], cherrypick=cherrypick,
                                   copyccs=cmdargs.copyccs)
    except KeyError:
        sys.exit(1)

logger.info('---')

if cherrypick is None:
        logger.critical('Total patches: %s', len(am_mbx))
    else:
        logger.info('Total patches: %s (cherrypicked: %s)', len(am_mbx), cmdargs.cherrypick)
    if lser.has_cover and lser.patches[0].followup_trailers and not covertrailers:
        # Warn that some trailers were sent to the cover letter
        logger.critical('---')
        logger.critical('NOTE: Some trailers were sent to the cover letter:')
        tseen = set()
        for trailer in lser.patches[0].followup_trailers:
            if tuple(trailer[:2]) not in tseen:
                logger.critical('      %s: %s', trailer[0], trailer[1])
                tseen.add(tuple(trailer[:2]))
        logger.critical('NOTE: Rerun with -t to apply them to all patches')
    if len(lser.trailer_mismatches):
        logger.critical('---')
        logger.critical('NOTE: some trailers ignored due to from/email mismatches:')
        for tname, tvalue, fname, femail in lser.trailer_mismatches:
            logger.critical('    ! Trailer: %s: %s', tname, tvalue)
            logger.critical('     Msg From: %s <%s>', fname, femail)
        logger.critical('NOTE: Rerun with -S to apply them anyway')

topdir = None
    # Are we in a git tree and if so, what is our toplevel?
    gitargs = ['rev-parse', '--show-toplevel']
    lines = b4.git_get_command_lines(None, gitargs)
    if len(lines) == 1:
        topdir = lines[0]

if cmdargs.threeway:
        if not topdir:
            logger.critical('WARNING: cannot prepare 3-way (not in a git dir)')
        elif not lser.complete:
            logger.critical('WARNING: cannot prepare 3-way (series incomplete)')
        else:
            rstart, rend = lser.make_fake_am_range(gitdir=None)
            if rstart and rend:
                logger.info('Prepared a fake commit range for 3-way merge (%.12s..%.12s)', rstart, rend)

logger.critical('---')
    if not lser.complete and not cmdargs.cherrypick:
        logger.critical('WARNING: Thread incomplete!')

if lser.has_cover and not cmdargs.nocover:
        lser.save_cover(am_cover)

top_msgid = None
    first_body = None
    for lmsg in lser.patches:
        if lmsg is not None:
            first_body = lmsg.body
            top_msgid = lmsg.msgid
            break
    if top_msgid is None:
        logger.critical('Could not find any patches in the series.')
        return

linkurl = config['linkmask'] % top_msgid
    if cmdargs.quiltready:
        q_dirname = os.path.join(outdir, '%s.patches' % slug)
        am_mbox_to_quilt(am_mbx, q_dirname)
        logger.critical('Quilt: %s', q_dirname)

logger.critical(' Link: %s', linkurl)

base_commit = None
    matches = re.search(r'base-commit: .*?([0-9a-f]+)', first_body, re.MULTILINE)
    if matches:
        base_commit = matches.groups()[0]
    else:
        # Try a more relaxed search
        matches = re.search(r'based on .*?([0-9a-f]{40})', first_body, re.MULTILINE)
        if matches:
            base_commit = matches.groups()[0]

if base_commit:
        logger.critical(' Base: %s', base_commit)
        logger.critical('       git checkout -b %s %s', gitbranch, base_commit)
        if cmdargs.outdir != '-':
            logger.critical('       git am %s', am_filename)
    else:
        cleanmsg = ''
        if topdir is not None:
            checked, mismatches = lser.check_applies_clean(topdir)
            if mismatches == 0 and checked != mismatches:
                cleanmsg = ' (applies clean to current tree)'
            elif cmdargs.guessbase:
                # Look at the last 10 tags and see if it applies cleanly to
                # any of them. I'm not sure how useful this is, but I'm going
                # to put it in for now and maybe remove later if it causes
                # problems or slowness
                if checked != mismatches:
                    best_matches = mismatches
                    cleanmsg = ' (best guess: current tree)'
                else:
                    best_matches = None
                # sort the tags by authordate
                gitargs = ['tag', '-l', '--sort=-taggerdate']
                lines = b4.git_get_command_lines(None, gitargs)
                if lines:
                    # Check last 10 tags
                    for tag in lines[:10]:
                        logger.debug('Checking base-commit possibility for %s', tag)
                        checked, mismatches = lser.check_applies_clean(topdir, tag)
                        if mismatches == 0 and checked != mismatches:
                            cleanmsg = ' (applies clean to: %s)' % tag
                            break
                        # did they all mismatch?
                        if checked == mismatches:
                            continue
                        if best_matches is None or mismatches < best_matches:
                            best_matches = mismatches
                            cleanmsg = ' (best guess: %s)' % tag

logger.critical(' Base: not found%s', cleanmsg)
        if cmdargs.outdir != '-':
            logger.critical('       git am %s', am_filename)

am_mbx.close()
    if cmdargs.outdir == '-':
        logger.info('---')
        with open(am_filename, 'rb') as fh:
            shutil.copyfileobj(fh, sys.stdout.buffer)
        os.unlink(am_filename)

thanks_record_am(lser, cherrypick=cherrypick)

def thanks_record_am(lser, cherrypick=None):
    # Are we tracking this already?
    datadir = b4.get_data_dir()
    slug = lser.get_slug(extended=True)
    filename = '%s.am' % slug

patches = list()
    at = 0
    padlen = len(str(lser.expected))
    lmsg = None

for pmsg in lser.patches:
        if pmsg is None:
            at += 1
            continue

if lmsg is None:
            lmsg = pmsg

if not pmsg.has_diff:
            # Don't care about the cover letter
            at += 1
            continue

if cherrypick is not None and at not in cherrypick:
            logger.debug('Skipped non-cherrypicked: %s', at)
            at += 1
            continue

pmsg.load_hashes()
        if pmsg.attestation is None:
            logger.debug('Unable to get hashes for all patches, not tracking for thanks')
            return

prefix = '%s/%s' % (str(pmsg.counter).zfill(padlen), pmsg.expected)
        patches.append((pmsg.subject, pmsg.pwhash, pmsg.msgid, prefix))
        at += 1

if lmsg is None:
        logger.debug('All patches missing, not tracking for thanks')
        return

allto = email.utils.getaddresses([str(x) for x in lmsg.msg.get_all('to', [])])
    allcc = email.utils.getaddresses([str(x) for x in lmsg.msg.get_all('cc', [])])

out = {
        'msgid': lmsg.msgid,
        'subject': lmsg.full_subject,
        'fromname': lmsg.fromname,
        'fromemail': lmsg.fromemail,
        'to': b4.format_addrs(allto, clean=False),
        'cc': b4.format_addrs(allcc, clean=False),
        'references': b4.LoreMessage.clean_header(lmsg.msg['References']),
        'sentdate': b4.LoreMessage.clean_header(lmsg.msg['Date']),
        'quote': b4.make_quote(lmsg.body, maxlines=5),
        'cherrypick': cherrypick is not None,
        'patches': patches,
    }
    fullpath = os.path.join(datadir, filename)
    with open(fullpath, 'w', encoding='utf-8') as fh:
        json.dump(out, fh, ensure_ascii=False, indent=4)
        logger.debug('Wrote %s for thanks tracking', filename)

def am_mbox_to_quilt(am_mbx, q_dirname):
    if os.path.exists(q_dirname):
        logger.critical('ERROR: Directory %s exists, not saving quilt patches', q_dirname)
        return
    os.mkdir(q_dirname, 0o755)
    patch_filenames = list()
    for key, msg in am_mbx.items():
        # Run each message through git mailinfo
        msg_out = mkstemp(suffix=None, prefix=None, dir=q_dirname)
        patch_out = mkstemp(suffix=None, prefix=None, dir=q_dirname)
        cmdargs = ['mailinfo', '--encoding=UTF-8', msg_out[1], patch_out[1]]
        ecode, info = b4.git_run_command(None, cmdargs, msg.as_bytes(policy=b4.emlpolicy))
        if not len(info.strip()):
            logger.critical('ERROR: Could not get mailinfo from patch %s', msg['Subject'])
            continue
        patchinfo = dict()
        for line in info.split('\n'):
            line = line.strip()
            if not line:
                continue
            chunks = line.split(':',  1)
            patchinfo[chunks[0]] = chunks[1]

slug = re.sub(r'\W+', '_', patchinfo['Subject']).strip('_').lower()
        patch_filename = '%04d_%s.patch' % (key+1, slug)
        patch_filenames.append(patch_filename)
        quilt_out = os.path.join(q_dirname, patch_filename)
        with open(quilt_out, 'wb') as fh:
            line = 'From: %s <%s>\n' % (patchinfo['Author'].strip(), patchinfo['Email'].strip())
            fh.write(line.encode('utf-8'))
            line = 'Subject: %s\n' % patchinfo['Subject'].strip()
            fh.write(line.encode('utf-8'))
            line = 'Date: %s\n' % patchinfo['Date'].strip()
            fh.write(line.encode('utf-8'))
            fh.write('\n'.encode('utf-8'))
            with open(msg_out[1], 'r') as mfh:
                fh.write(mfh.read().encode('utf-8'))
            with open(patch_out[1], 'r') as pfh:
                fh.write(pfh.read().encode('utf-8'))
        logger.debug('  Wrote: %s', patch_filename)
        os.unlink(msg_out[1])
        os.unlink(patch_out[1])
    # Write the series file
    with open(os.path.join(q_dirname, 'series'), 'w') as sfh:
        for patch_filename in patch_filenames:
            sfh.write('%s\n' % patch_filename)

def get_extra_series(mboxfile, direction=1, wantvers=None, nocache=False):
    # Open the mbox and find the latest series mentioned in it
    if os.path.isdir(mboxfile):
        mbx = mailbox.Maildir(mboxfile)
    else:
        mbx = mailbox.mbox(mboxfile)

base_msg = None
    latest_revision = None
    seen_msgids = list()
    seen_covers = list()
    for key, msg in mbx.items():
        msgid = b4.LoreMessage.get_clean_msgid(msg)
        seen_msgids.append(msgid)
        lsub = b4.LoreSubject(msg['Subject'])
        # Ignore replies or counters above 1
        if lsub.reply or lsub.counter > 1:
            continue
        if base_msg is not None:
            logger.debug('Current base_msg: %s', base_msg['Subject'])
        logger.debug('Checking the subject on %s', lsub.full_subject)
        if latest_revision is None or lsub.revision >= latest_revision:
            latest_revision = lsub.revision
            if lsub.counter == 0 and not lsub.counters_inferred:
                # And a cover letter, nice. This is the easy case
                base_msg = msg
                seen_covers.append(latest_revision)
            elif lsub.counter == 1 and latest_revision not in seen_covers:
                # A patch/series without a cover letter
                base_msg = msg

if base_msg is None:
        logger.debug('Could not find cover of 1st patch in mbox')
        mbx.close()
        return
    # Get subject info from base_msg again
    lsub = b4.LoreSubject(base_msg['Subject'])
    if not len(lsub.prefixes):
        logger.debug('Not checking for new revisions: no prefixes on the cover letter.')
        mbx.close()
        return
    if direction < 0 and latest_revision <= 1:
        logger.debug('This is the latest version of the series')
        mbx.close()
        return
    if direction < 0 and wantvers is None:
        wantvers = [latest_revision - 1]

base_msgid = b4.LoreMessage.get_clean_msgid(base_msg)
    fromeml = email.utils.getaddresses(base_msg.get_all('from', []))[0][1]
    msgdate = email.utils.parsedate_tz(str(base_msg['Date']))
    startdate = time.strftime('%Y%m%d', msgdate[:9])
    listarc = base_msg.get_all('List-Archive')[-1].strip('<>')
    if direction > 0:
        q = 's:"%s" AND f:"%s" AND d:%s..' % (lsub.subject.replace('"', ''), fromeml, startdate)
        queryurl = '%s?%s' % (listarc, urllib.parse.urlencode({'q': q, 'x': 'A', 'o': '-1'}))
        logger.critical('Checking for newer revisions on %s', listarc)
    else:
        q = 's:"%s" AND f:"%s" AND d:..%s' % (lsub.subject.replace('"', ''), fromeml, startdate)
        queryurl = '%s?%s' % (listarc, urllib.parse.urlencode({'q': q, 'x': 'A', 'o': '1'}))
        logger.critical('Checking for older revisions on %s', listarc)

logger.debug('Query URL: %s', queryurl)
    session = b4.get_requests_session()
    resp = session.get(queryurl)
    # try to parse it
    try:
        tree = xml.etree.ElementTree.fromstring(resp.content)
    except xml.etree.ElementTree.ParseError as ex:
        logger.debug('Unable to parse results, ignoring: %s', ex)
        resp.close()
        mbx.close()
        return
    resp.close()
    ns = {'atom': 'http://www.w3.org/2005/Atom'}
    entries = tree.findall('atom:entry', ns)

for entry in entries:
        title = entry.find('atom:title', ns).text
        lsub = b4.LoreSubject(title)
        if lsub.reply or lsub.counter > 1:
            logger.debug('Ignoring result (not interesting): %s', title)
            continue
        link = entry.find('atom:link', ns).get('href')
        if direction > 0 and lsub.revision <= latest_revision:
            logger.debug('Ignoring result (not new revision): %s', title)
            continue
        elif direction < 0 and lsub.revision >= latest_revision:
            logger.debug('Ignoring result (not old revision): %s', title)
            continue
        elif direction < 0 and lsub.revision not in wantvers:
            logger.debug('Ignoring result (not revision we want): %s', title)
            continue
        if link.find('/%s/' % base_msgid) > 0:
            logger.debug('Ignoring result (same thread as ours):%s', title)
            continue
        if lsub.revision == 1 and lsub.revision == latest_revision:
            # Someone sent a separate message with an identical title but no new vX in the subject line
            if direction > 0:
                # It's *probably* a new revision.
                logger.debug('Likely a new revision: %s', title)
            else:
                # It's *probably* an older revision.
                logger.debug('Likely an older revision: %s', title)
        elif direction > 0 and lsub.revision > latest_revision:
            logger.debug('Definitely a new revision [v%s]: %s', lsub.revision, title)
        elif direction < 0 and lsub.revision < latest_revision:
            logger.debug('Definitely an older revision [v%s]: %s', lsub.revision, title)
        else:
            logger.debug('No idea what this is: %s', title)
            continue
        t_mbx_url = '%st.mbox.gz' % link
        savefile = mkstemp('b4-get')[1]
        nt_mboxfile = b4.get_pi_thread_by_url(t_mbx_url, savefile, nocache=nocache)
        nt_mbx = mailbox.mbox(nt_mboxfile)
        # Append all of these to the existing mailbox
        new_adds = 0
        for nt_msg in nt_mbx:
            nt_msgid = b4.LoreMessage.get_clean_msgid(nt_msg)
            if nt_msgid in seen_msgids:
                logger.debug('Duplicate message, skipping')
                continue
            nt_subject = re.sub(r'\s+', ' ', nt_msg['Subject'])
            logger.debug('Adding: %s', nt_subject)
            new_adds += 1
            mbx.add(nt_msg)
            seen_msgids.append(nt_msgid)
        nt_mbx.close()
        if new_adds:
            logger.info('Added %s messages from thread: %s', new_adds, title)
        logger.debug('Removing temporary %s', nt_mboxfile)
        os.unlink(nt_mboxfile)

# We close the mbox, since we'll be reopening it later
    mbx.close()

def main(cmdargs):
    if cmdargs.checknewer:
        # Force nocache mode
        cmdargs.nocache = True

savefile = mkstemp('b4-mbox')[1]

if not cmdargs.localmbox:
        msgid = b4.get_msgid(cmdargs)

threadfile = b4.get_pi_thread_by_msgid(msgid, savefile, useproject=cmdargs.useproject, nocache=cmdargs.nocache)
        if threadfile is None:
            os.unlink(savefile)
            return
    else:
        if os.path.exists(cmdargs.localmbox):
            msgid = b4.get_msgid(cmdargs)
            if os.path.isdir(cmdargs.localmbox):
                in_mbx = mailbox.Maildir(cmdargs.localmbox)
            else:
                in_mbx = mailbox.mbox(cmdargs.localmbox)
            out_mbx = mailbox.mbox(savefile)
            b4.save_strict_thread(in_mbx, out_mbx, msgid)
            if not len(out_mbx):
                logger.critical('Could not find %s in %s', msgid, cmdargs.localmbox)
                os.unlink(savefile)
                sys.exit(1)
            threadfile = savefile
        else:
            logger.critical('Mailbox %s does not exist', cmdargs.localmbox)
            os.unlink(savefile)
            sys.exit(1)

if threadfile and cmdargs.checknewer:
        get_extra_series(threadfile, direction=1)

if cmdargs.subcmd == 'am':
        mbox_to_am(threadfile, cmdargs)
        os.unlink(threadfile)
        return

mbx = mailbox.mbox(threadfile)
    logger.info('%s messages in the thread', len(mbx))
    if cmdargs.outdir == '-':
        mbx.close()
        logger.info('---')
        with open(threadfile, 'rb') as fh:
            shutil.copyfileobj(fh, sys.stdout.buffer)
        os.unlink(threadfile)
        return

# Check if outdir is a maildir
    if (os.path.isdir(os.path.join(cmdargs.outdir, 'new'))
            and os.path.isdir(os.path.join(cmdargs.outdir, 'cur'))
            and os.path.isdir(os.path.join(cmdargs.outdir, 'tmp'))):
        mdr = mailbox.Maildir(cmdargs.outdir)
        have_msgids = set()
        added = 0
        if cmdargs.filterdupes:
            for emsg in mdr:
                have_msgids.add(b4.LoreMessage.get_clean_msgid(emsg))
        for msg in mbx:
            if b4.LoreMessage.get_clean_msgid(msg) not in have_msgids:
                added += 1
                mdr.add(msg)
        logger.info('Added to %s messages to maildir %s', added, cmdargs.outdir)
        mbx.close()
        os.unlink(threadfile)
        return

if cmdargs.wantname:
        savefile = os.path.join(cmdargs.outdir, cmdargs.wantname)
    else:
        msgid = b4.get_msgid(cmdargs)
        savefile = os.path.join(cmdargs.outdir, '%s.mbx' % msgid)

mbx.close()
    shutil.copy(threadfile, savefile)
    logger.info('Saved %s', savefile)
    os.unlink(threadfile)
0707010000000A000081A40000273B00000064000000015FEA205100003529000000000000000000000000000000000000001400000000b4-0.6.2+4/b4/pr.py#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright (C) 2020 by the Linux Foundation
#
__author__ = 'Konstantin Ryabitsev <konstantin@linuxfoundation.org>'

import os
import sys
import b4
import re
import mailbox
import json

from datetime import timedelta
from tempfile import mkstemp
from email import utils, charset

charset.add_charset('utf-8', None)

logger = b4.logger

PULL_BODY_SINCE_ID_RE = [
    re.compile(r'changes since commit ([0-9a-f]{5,40}):', re.M | re.I)
]

# I like these
PULL_BODY_WITH_COMMIT_ID_RE = [
    re.compile(r'fetch changes up to ([0-9a-f]{5,40}):', re.M | re.I),
]

# I don't like these
PULL_BODY_REMOTE_REF_RE = [
    re.compile(r'^\s*([\w+-]+(?:://|@)[\w/.@:~-]+)[\s\\]+([\w/._-]+)\s*$', re.M | re.I),
    re.compile(r'^\s*([\w+-]+(?:://|@)[\w/.@~-]+)\s*$', re.M | re.I),
]

def git_get_commit_id_from_repo_ref(repo, ref):
    # We only handle git and http/s URLs
    if not (repo.find('git://') == 0 or repo.find('http://') == 0 or repo.find('https://') == 0):
        logger.debug('%s uses unsupported protocol', repo)
        return None

logger.debug('getting commit-id from: %s %s', repo, ref)
    # Drop the leading "refs/", if any
    ref = re.sub(r'^refs/', '', ref)
    # Is it a full ref name or a shortname?
    if ref.find('heads/') < 0 and ref.find('tags/') < 0:
        # Try grabbing it as a head first
        lines = b4.git_get_command_lines(None, ['ls-remote', repo, 'refs/heads/%s' % ref])
        if not lines:
            # try it as a tag, then
            lines = b4.git_get_command_lines(None, ['ls-remote', repo, 'refs/tags/%s^{}' % ref])

elif ref.find('tags/') == 0:
        lines = b4.git_get_command_lines(None, ['ls-remote', repo, 'refs/%s^{}' % ref])

else:
        # Grab it as a head and hope for the best
        lines = b4.git_get_command_lines(None, ['ls-remote', repo, 'refs/%s' % ref])

if not lines:
        # Oh well, we tried
        logger.debug('did not find commit-id, ignoring pull request')
        return None

commit_id = lines[0].split()[0]
    logger.debug('success, commit-id: %s', commit_id)
    return commit_id

def parse_pr_data(msg):
    lmsg = b4.LoreMessage(msg)
    if lmsg.body is None:
        logger.critical('Could not find a plain part in the message body')
        return None

logger.info('Looking at: %s', lmsg.full_subject)

for since_re in PULL_BODY_SINCE_ID_RE:
        matches = since_re.search(lmsg.body)
        if matches:
            lmsg.pr_base_commit = matches.groups()[0]
            break

for reporef_re in PULL_BODY_REMOTE_REF_RE:
        matches = reporef_re.search(lmsg.body)
        if matches:
            chunks = matches.groups()
            lmsg.pr_repo = chunks[0]
            if len(chunks) > 1:
                lmsg.pr_ref = chunks[1]
            else:
                lmsg.pr_ref = 'refs/heads/master'
            break

for cid_re in PULL_BODY_WITH_COMMIT_ID_RE:
        matches = cid_re.search(lmsg.body)
        if matches:
            lmsg.pr_tip_commit = matches.groups()[0]
            break

if lmsg.pr_repo and lmsg.pr_ref:
        lmsg.pr_remote_tip_commit = git_get_commit_id_from_repo_ref(lmsg.pr_repo, lmsg.pr_ref)

return lmsg

def attest_fetch_head(gitdir, lmsg):
    config = b4.get_main_config()
    attpolicy = config['attestation-policy']
    if config['attestation-checkmarks'] == 'fancy':
        attpass = b4.PASS_FANCY
        attfail = b4.FAIL_FANCY
    else:
        attpass = b4.PASS_SIMPLE
        attfail = b4.FAIL_SIMPLE
    # Is FETCH_HEAD a tag or a commit?
    htype = b4.git_get_command_lines(gitdir, ['cat-file', '-t', 'FETCH_HEAD'])
    passing = False
    out = ''
    otype = 'unknown'
    if len(htype):
        otype = htype[0]
    if otype == 'tag':
        ecode, out = b4.git_run_command(gitdir, ['verify-tag', '--raw', 'FETCH_HEAD'], logstderr=True)
    elif otype == 'commit':
        ecode, out = b4.git_run_command(gitdir, ['verify-commit', '--raw', 'FETCH_HEAD'], logstderr=True)

good, valid, trusted, attestor, sigdate, errors = b4.validate_gpg_signature(out, 'pgp')

if good and valid and trusted:
        passing = True

out = out.strip()
    if not len(out) and attpolicy != 'check':
        errors.add('Remote %s is not signed!' % otype)

if passing:
        trailer = attestor.get_trailer(lmsg.fromemail)
        logger.info('  ---')
        logger.info('  %s %s', attpass, trailer)
        return

if errors:
        logger.critical('  ---')
        if len(out):
            logger.critical('  Pull request is signed, but verification did not succeed:')
        else:
            logger.critical('  Pull request verification did not succeed:')
        for error in errors:
            logger.critical('    %s %s', attfail, error)

if attpolicy == 'hardfail':
            import sys
            sys.exit(128)

def fetch_remote(gitdir, lmsg, branch=None):
    # Do we know anything about this base commit?
    if lmsg.pr_base_commit and not b4.git_commit_exists(gitdir, lmsg.pr_base_commit):
        logger.critical('ERROR: git knows nothing about commit %s', lmsg.pr_base_commit)
        logger.critical('       Are you running inside a git checkout and is it up-to-date?')
        return 1

if lmsg.pr_tip_commit != lmsg.pr_remote_tip_commit:
        logger.critical('ERROR: commit-id mismatch between pull request and remote')
        logger.critical('       msg=%s, remote=%s', lmsg.pr_tip_commit, lmsg.pr_remote_tip_commit)
        return 1

# Fetch it now
    logger.info('  Fetching %s %s', lmsg.pr_repo, lmsg.pr_ref)
    gitargs = ['fetch', lmsg.pr_repo, lmsg.pr_ref]
    ecode, out = b4.git_run_command(gitdir, gitargs, logstderr=True)
    if ecode > 0:
        logger.critical('ERROR: Could not fetch remote:')
        logger.critical(out)
        return ecode

config = b4.get_main_config()
    if config['attestation-policy'] != 'off':
        attest_fetch_head(gitdir, lmsg)

logger.info('---')
    if branch:
        gitargs = ['checkout', '-b', branch, 'FETCH_HEAD']
        logger.info('Fetched into branch %s', branch)
        ecode, out = b4.git_run_command(gitdir, gitargs)
        if ecode > 0:
            logger.critical('ERROR: Failed to create branch')
            logger.critical(out)
            return ecode
    else:
        logger.info('Successfully fetched into FETCH_HEAD')

thanks_record_pr(lmsg)

return 0

def thanks_record_pr(lmsg):
    datadir = b4.get_data_dir()
    # Check if we're tracking it already
    filename = '%s.pr' % lmsg.pr_remote_tip_commit
    for entry in os.listdir(datadir):
        if entry == filename:
            return
    allto = utils.getaddresses([str(x) for x in lmsg.msg.get_all('to', [])])
    allcc = utils.getaddresses([str(x) for x in lmsg.msg.get_all('cc', [])])
    out = {
        'msgid': lmsg.msgid,
        'subject': lmsg.full_subject,
        'fromname': lmsg.fromname,
        'fromemail': lmsg.fromemail,
        'to': b4.format_addrs(allto, clean=False),
        'cc': b4.format_addrs(allcc, clean=False),
        'references': b4.LoreMessage.clean_header(lmsg.msg['References']),
        'remote': lmsg.pr_repo,
        'ref': lmsg.pr_ref,
        'sentdate': b4.LoreMessage.clean_header(lmsg.msg['Date']),
        'quote': b4.make_quote(lmsg.body, maxlines=6)
    }
    fullpath = os.path.join(datadir, filename)
    with open(fullpath, 'w', encoding='utf-8') as fh:
        json.dump(out, fh, ensure_ascii=False, indent=4)
        logger.debug('Wrote %s for thanks tracking', filename)

def explode(gitdir, lmsg, savefile):
    # We always fetch into FETCH_HEAD when exploding
    ecode = fetch_remote(gitdir, lmsg)
    if ecode > 0:
        sys.exit(ecode)
    if not lmsg.pr_base_commit:
        # Use git merge-base between HEAD and FETCH_HEAD to find
        # where we should start
        logger.info('Running git merge-base to find common ancestry')
        gitargs = ['merge-base', 'HEAD', 'FETCH_HEAD']
        ecode, out = b4.git_run_command(gitdir, gitargs, logstderr=True)
        if ecode > 0:
            logger.critical('Could not find common ancestry.')
            logger.critical(out)
            sys.exit(ecode)
        lmsg.pr_base_commit = out.strip()
    logger.info('Generating patches starting from the base-commit')
    reroll = None
    if lmsg.revision > 1:
        reroll = lmsg.revision
    ecode, out = b4.git_format_patches(gitdir, lmsg.pr_base_commit, 'FETCH_HEAD', reroll=reroll)
    if ecode > 0:
        logger.critical('ERROR: Could not convert pull request into patches')
        logger.critical(out)
        sys.exit(ecode)

# Fix From lines to make sure this is a valid mboxo
    out = re.sub(r'^From (?![a-f0-9]+ \w+ \w+ \d+ \d+:\d+:\d+ \d+$)', '>From ', out, 0, re.M)

# Save patches into a temporary file
    patchmbx = mkstemp()[1]
    with open(patchmbx, 'w') as fh:
        fh.write(out)
    pmbx = mailbox.mbox(patchmbx)
    embx = mailbox.mbox(savefile)
    cover = lmsg.get_am_message()
    # Add base-commit to the cover
    body = cover.get_payload(decode=True)
    body = '%s\nbase-commit: %s\n' % (body.decode('utf-8'), lmsg.pr_base_commit)
    cover.set_payload(body)
    bout = cover.as_string(policy=b4.emlpolicy)
    embx.add(bout.encode('utf-8'))

# Set the pull request message as cover letter
    for msg in pmbx:
        # Move the original From and Date into the body
        prepend = list()
        if msg['from'] != lmsg.msg['from']:
            cleanfrom = b4.LoreMessage.clean_header(msg['from'])
            prepend.append('From: %s' % ''.join(cleanfrom))
        prepend.append('Date: %s' % msg['date'])
        body = '%s\n\n%s' % ('\n'.join(prepend), msg.get_payload(decode=True).decode('utf-8'))
        msg.set_payload(body)
        msubj = b4.LoreSubject(msg['subject'])
        msg.replace_header('Subject', msubj.full_subject)
        # Set from, to, cc, date headers to match the original pull request
        msg.replace_header('From', b4.LoreMessage.clean_header(lmsg.msg['From']))
        # Add a number of seconds equalling the counter, in hopes it gets properly threaded
        newdate = lmsg.date + timedelta(seconds=msubj.counter)
        msg.replace_header('Date', utils.format_datetime(newdate))
        msg.add_header('To', b4.LoreMessage.clean_header(lmsg.msg['To']))
        if lmsg.msg['Cc']:
            msg.add_header('Cc', b4.LoreMessage.clean_header(lmsg.msg['Cc']))
        # Set the message-id based on the original pull request msgid
        msg.add_header('Message-Id', '<b4-exploded-%s-%s>' % (msubj.counter, lmsg.msgid))
        msg.add_header('In-Reply-To', '<%s>' % lmsg.msgid)
        if lmsg.msg['References']:
            msg.add_header('References', '%s <%s>' % (
                b4.LoreMessage.clean_header(lmsg.msg['References']), lmsg.msgid))
        else:
            msg.add_header('References', '<%s>' % lmsg.msgid)
        if lmsg.msg['List-Id']:
            msg.add_header('List-Id', b4.LoreMessage.clean_header(lmsg.msg['List-Id']))
        msg.add_header('X-Mailer', 'b4-explode/%s' % b4.__VERSION__)
        logger.info('  %s', msubj.full_subject)
        msg.set_charset('utf-8')
        bout = msg.as_string(policy=b4.emlpolicy)
        embx.add(bout.encode('utf-8'))
    logger.info('---')
    logger.info('Wrote %s patches into %s', len(pmbx), savefile)
    pmbx.close()
    os.unlink(patchmbx)
    embx.close()
    sys.exit(0)

def main(cmdargs):
    gitdir = cmdargs.gitdir

msgid = b4.get_msgid(cmdargs)
    savefile = mkstemp()[1]
    mboxfile = b4.get_pi_thread_by_msgid(msgid, savefile)
    if mboxfile is None:
        os.unlink(savefile)
        return
    # Find the message with the msgid we were asked about
    mbx = mailbox.mbox(mboxfile)
    lmsg = None
    for msg in mbx:
        mmsgid = b4.LoreMessage.get_clean_msgid(msg)
        if mmsgid == msgid:
            lmsg = parse_pr_data(msg)

# Got all we need from it
    mbx.close()
    os.unlink(savefile)

if lmsg is None or lmsg.pr_remote_tip_commit is None:
        logger.critical('ERROR: Could not find pull request info in %s', msgid)
        sys.exit(1)

if not lmsg.pr_tip_commit:
        lmsg.pr_tip_commit = lmsg.pr_remote_tip_commit

if cmdargs.explode:
        savefile = cmdargs.outmbox
        if savefile is None:
            savefile = '%s.mbx' % lmsg.msgid
        if os.path.exists(savefile):
            logger.info('File exists: %s', savefile)
            sys.exit(1)
        explode(gitdir, lmsg, savefile)

exists = b4.git_commit_exists(gitdir, lmsg.pr_tip_commit)

if exists:
        # Is it in any branch, or just flapping in the wind?
        branches = b4.git_branch_contains(gitdir, lmsg.pr_tip_commit)
        if len(branches):
            logger.info('Pull request tip commit exists in the following branches:')
            for branch in branches:
                logger.info('  %s', branch)
            if cmdargs.check:
                sys.exit(0)
            sys.exit(1)

# Is it at the tip of FETCH_HEAD?
        loglines = b4.git_get_command_lines(gitdir, ['log', '-1', '--pretty=oneline', 'FETCH_HEAD'])
        if len(loglines) and loglines[0].find(lmsg.pr_tip_commit) == 0:
            logger.info('Pull request is at the tip of FETCH_HEAD')
            if cmdargs.check:
                attest_fetch_head(gitdir, lmsg)
                sys.exit(0)

elif cmdargs.check:
        logger.info('Pull request does not appear to be in this tree.')
        sys.exit(0)

fetch_remote(gitdir, lmsg, branch=cmdargs.branch)
0707010000000B000081A40000273B00000064000000015FEA205100005647000000000000000000000000000000000000001400000000b4-0.6.2+4/b4/ty.py#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# SPDX-License-Identifier: GPL-2.0-or-later
# Copyright (C) 2020 by the Linux Foundation
#
__author__ = 'Konstantin Ryabitsev <konstantin@linuxfoundation.org>'

import os
import sys
import b4
import re
import email
import email.message
import json

from string import Template
from email import utils
from pathlib import Path

logger = b4.logger

DEFAULT_PR_TEMPLATE = """
On ${sentdate}, ${fromname} wrote:
${quote}

Merged, thanks!

${summary}

Best regards,
-- 
${signature}
"""

DEFAULT_AM_TEMPLATE = """
On ${sentdate}, ${fromname} wrote:
${quote}

Applied, thanks!

${summary}

Best regards,
-- 
${signature}
"""

# Used to track commits created by current user
MY_COMMITS = None
# Used to track additional branch info
BRANCH_INFO = None

def git_get_merge_id(gitdir, commit_id, branch=None):
    # get merge commit id
    args = ['rev-list', '%s..' % commit_id, '--ancestry-path']
    if branch is not None:
        args += [branch]
    lines = b4.git_get_command_lines(gitdir, args)
    if not len(lines):
        return None
    return lines[-1]

def git_get_rev_diff(gitdir, rev):
    args = ['diff', '%s~..%s' % (rev, rev)]
    return b4.git_run_command(gitdir, args)

def git_get_commit_message(gitdir, rev):
    args = ['log', '--format=%B', '-1', rev]
    return b4.git_run_command(gitdir, args)

def make_reply(reply_template, jsondata):
    body = Template(reply_template).safe_substitute(jsondata)
    # Conform to email standards
    body = body.replace('\n', '\r\n')
    msg = email.message_from_string(body)
    msg['From'] = '%s <%s>' % (jsondata['myname'], jsondata['myemail'])
    allto = utils.getaddresses([jsondata['to']])
    allcc = utils.getaddresses([jsondata['cc']])
    # Remove ourselves and original sender from allto or allcc
    for entry in list(allto):
        if entry[1] == jsondata['myemail'] or entry[1] == jsondata['fromemail']:
            allto.remove(entry)
    for entry in list(allcc):
        if entry[1] == jsondata['myemail'] or entry[1] == jsondata['fromemail']:
            allcc.remove(entry)

# Add original sender to the To
    allto.append((jsondata['fromname'], jsondata['fromemail']))

msg['To'] = b4.format_addrs(allto)
    if allcc:
        msg['Cc'] = b4.format_addrs(allcc)
    msg['In-Reply-To'] = '<%s>' % jsondata['msgid']
    if len(jsondata['references']):
        msg['References'] = '%s <%s>' % (jsondata['references'], jsondata['msgid'])
    else:
        msg['References'] = '<%s>' % jsondata['msgid']

subject = re.sub(r'^Re:\s+', '', jsondata['subject'], flags=re.I)
    if jsondata.get('cherrypick'):
        msg['Subject'] = 'Re: (subset) ' + subject
    else:
        msg['Subject'] = 'Re: ' + subject

mydomain = jsondata['myemail'].split('@')[1]
    msg['Message-Id'] = email.utils.make_msgid(idstring='b4-ty', domain=mydomain)
    msg['Date'] = email.utils.formatdate(localtime=True)
    return msg

def auto_locate_pr(gitdir, jsondata, branch):
    pr_commit_id = jsondata['pr_commit_id']
    logger.debug('Checking %s', jsondata['pr_commit_id'])
    if not b4.git_commit_exists(gitdir, pr_commit_id):
        return None

onbranches = b4.git_branch_contains(gitdir, pr_commit_id)
    if not len(onbranches):
        logger.debug('%s is not on any branches', pr_commit_id)
        return None
    if branch not in onbranches:
        logger.debug('%s is not on branch %s', pr_commit_id, branch)
        return None

# Get the merge commit
    merge_commit_id = git_get_merge_id(gitdir, pr_commit_id, branch)
    if not merge_commit_id:
        logger.debug('Could not get a merge commit-id for %s', pr_commit_id)
        return None

# Check that we are the author of the merge commit
    gitargs = ['show', '--format=%ae', merge_commit_id]
    out = b4.git_get_command_lines(gitdir, gitargs)
    if not out:
        logger.debug('Could not get merge commit author for %s', pr_commit_id)
        return None

usercfg = b4.get_user_config()
    if usercfg['email'] not in out:
        logger.debug('Merged by a different author, ignoring %s', pr_commit_id)
        logger.debug('Author: %s', out[0])
        return None

return merge_commit_id

def get_all_commits(gitdir, branch, since='1.week', committer=None):
    global MY_COMMITS
    if MY_COMMITS is not None:
        return MY_COMMITS

MY_COMMITS = dict()
    if committer is None:
        usercfg = b4.get_user_config()
        committer = usercfg['email']

gitargs = ['log', '--committer', committer, '--no-abbrev', '--oneline', '--since', since, branch]
    lines = b4.git_get_command_lines(gitdir, gitargs)
    if not len(lines):
        logger.debug('No new commits from the current user --since=%s', since)
        return MY_COMMITS

logger.info('Found %s of your commits since %s', len(lines), since)
    logger.info('Calculating patch hashes, may take a moment...')
    # Get patch hash of each commit
    for line in lines:
        commit_id, subject = line.split(maxsplit=1)
        ecode, out = git_get_rev_diff(gitdir, commit_id)
        pwhash = b4.LoreMessage.get_patchwork_hash(out)
        logger.debug('phash=%s', pwhash)
        # get all message-id or link trailers
        ecode, out = git_get_commit_message(gitdir, commit_id)
        matches = re.findall(r'^\s*(?:message-id|link):[ \t]+(\S+)\s*$', out, flags=re.I | re.M)
        trackers = list()
        if matches:
            for tvalue in matches:
                trackers.append(tvalue)

MY_COMMITS[pwhash] = (commit_id, subject, trackers)

return MY_COMMITS

def auto_locate_series(gitdir, jsondata, branch, since='1.week'):
    commits = get_all_commits(gitdir, branch, since)

patchids = set(commits.keys())
    # We need to find all of them in the commits
    found = list()
    matches = 0
    at = 0
    for patch in jsondata['patches']:
        at += 1
        logger.debug('Checking %s', patch)
        if patch[1] in patchids:
            logger.debug('Found: %s', patch[0])
            found.append((at, commits[patch[1]][0]))
            matches += 1
        else:
            # try to locate by subject
            success = False
            for pwhash, commit in commits.items():
                if commit[1] == patch[0]:
                    logger.debug('Matched using subject')
                    found.append((at, commit[0]))
                    success = True
                    matches += 1
                    break
                elif len(patch) > 2 and len(patch[2]) and len(commit[2]):
                    for tracker in commit[2]:
                        if tracker.find(patch[2]) >= 0:
                            logger.debug('Matched using recorded message-id')
                            found.append((at, commit[0]))
                            success = True
                            matches += 1
                            break
                if success:
                    break

if not success:
                logger.debug('  Failed to find a match for: %s', patch[0])
                found.append((at, None))

return found

def read_template(tptfile):
    # bubbles up FileNotFound
    tpt = ''
    if tptfile.find('~') >= 0:
        tptfile = os.path.expanduser(tptfile)
    if tptfile.find('$') >= 0:
        tptfile = os.path.expandvars(tptfile)
    with open(tptfile, 'r', encoding='utf-8') as fh:
        for line in fh:
            if len(line) and line[0] == '#':
                continue
            tpt += line
    return tpt

def set_branch_details(gitdir, branch, jsondata, config):
    binfo = get_branch_info(gitdir, branch)
    jsondata['branch'] = branch
    for key, val in binfo.items():
        if key == 'b4-treename':
            config['thanks-treename'] = val
        elif key == 'b4-commit-url-mask':
            config['thanks-commit-url-mask'] = val
        elif key == 'b4-pr-template':
            config['thanks-pr-template'] = val
        elif key == 'b4-am-template':
            config['thanks-am-template'] = val
        elif key == 'branch':
            jsondata['branch'] = val

if 'thanks-treename' in config:
        jsondata['treename'] = config['thanks-treename']
    elif 'url' in binfo:
        # noinspection PyBroadException
        try:
            # Try to grab the last two chunks of the path
            purl = Path(binfo['url'])
            jsondata['treename'] = os.path.join(purl.parts[-2], purl.parts[-1])
        except:
            # Something went wrong... just use the whole URL
            jsondata['treename'] = binfo['url']
    else:
        jsondata['treename'] = 'local tree'

return jsondata, config

def generate_pr_thanks(gitdir, jsondata, branch):
    config = b4.get_main_config()
    jsondata, config = set_branch_details(gitdir, branch, jsondata, config)
    thanks_template = DEFAULT_PR_TEMPLATE
    if config['thanks-pr-template']:
        # Try to load this template instead
        try:
            thanks_template = read_template(config['thanks-pr-template'])
        except FileNotFoundError:
            logger.critical('ERROR: thanks-pr-template says to use %s, but it does not exist',
                            config['thanks-pr-template'])
            sys.exit(2)

if 'merge_commit_id' not in jsondata:
        merge_commit_id = git_get_merge_id(gitdir, jsondata['pr_commit_id'])
        if not merge_commit_id:
            logger.critical('Could not get merge commit id for %s', jsondata['subject'])
            logger.critical('Was it actually merged?')
            sys.exit(1)
        jsondata['merge_commit_id'] = merge_commit_id
    # Make a summary
    cidmask = config['thanks-commit-url-mask']
    if not cidmask:
        cidmask = 'merge commit: %s'
    jsondata['summary'] = cidmask % jsondata['merge_commit_id']
    msg = make_reply(thanks_template, jsondata)
    return msg

def generate_am_thanks(gitdir, jsondata, branch, since):
    config = b4.get_main_config()
    jsondata, config = set_branch_details(gitdir, branch, jsondata, config)
    thanks_template = DEFAULT_AM_TEMPLATE
    if config['thanks-am-template']:
        # Try to load this template instead
        try:
            thanks_template = read_template(config['thanks-am-template'])
        except FileNotFoundError:
            logger.critical('ERROR: thanks-am-template says to use %s, but it does not exist',
                            config['thanks-am-template'])
            sys.exit(2)
    if 'commits' not in jsondata:
        commits = auto_locate_series(gitdir, jsondata, branch, since)
    else:
        commits = jsondata['commits']

cidmask = config['thanks-commit-url-mask']
    if not cidmask:
        cidmask = 'commit: %s'
    slines = list()
    nomatch = 0
    padlen = len(str(len(commits)))
    patches = jsondata['patches']
    for at, cid in commits:
        try:
            prefix = '[%s] ' % patches[at-1][3]
        except IndexError:
            prefix = '[%s/%s] ' % (str(at).zfill(padlen), len(commits))
        slines.append('%s%s' % (prefix, patches[at-1][0]))
        if cid is None:
            slines.append('%s(no commit info)' % (' ' * len(prefix)))
            nomatch += 1
        else:
            slines.append('%s%s' % (' ' * len(prefix), cidmask % cid))
    jsondata['summary'] = '\n'.join(slines)
    if nomatch == len(commits):
        logger.critical('  WARNING: None of the patches matched for: %s', jsondata['subject'])
        logger.critical('           Please review the resulting message')
    elif nomatch > 0:
        logger.critical('  WARNING: Could not match %s of %s patches in: %s',
                        nomatch, len(commits), jsondata['subject'])
        logger.critical('           Please review the resulting message')

msg = make_reply(thanks_template, jsondata)
    return msg

def auto_thankanator(cmdargs):
    gitdir = cmdargs.gitdir
    wantbranch = get_wanted_branch(cmdargs)
    logger.info('Auto-thankanating commits in %s', wantbranch)
    tracked = list_tracked()
    if not len(tracked):
        logger.info('Nothing to do')
        sys.exit(0)

applied = list()
    for jsondata in tracked:
        if 'pr_commit_id' in jsondata:
            # this is a pull request
            merge_commit_id = auto_locate_pr(gitdir, jsondata, wantbranch)
            if merge_commit_id is None:
                continue
            jsondata['merge_commit_id'] = merge_commit_id
        else:
            # This is a patch series
            commits = auto_locate_series(gitdir, jsondata, wantbranch, since=cmdargs.since)
            # Weed out series that have no matches at all
            found = False
            for commit in commits:
                if commit[1] is not None:
                    found = True
                    break
            if not found:
                continue
            jsondata['commits'] = commits
        applied.append(jsondata)
        logger.info('  Located: %s', jsondata['subject'])

if not len(applied):
        logger.info('Nothing to do')
        sys.exit(0)

logger.info('---')
    send_messages(applied, cmdargs.gitdir, cmdargs.outdir, wantbranch, since=cmdargs.since)
    sys.exit(0)

def send_messages(listing, gitdir, outdir, branch, since='1.week'):
    # Not really sending, but writing them out to be sent on your own
    # We'll probably gain ability to send these once the feature is
    # more mature and we're less likely to mess things up
    datadir = b4.get_data_dir()
    logger.info('Generating %s thank-you letters', len(listing))
    # Check if the outdir exists and if it has any .thanks files in it
    if not os.path.exists(outdir):
        os.mkdir(outdir)

usercfg = b4.get_user_config()
    # Do we have a .signature file?
    sigfile = os.path.join(str(Path.home()), '.signature')
    if os.path.exists(sigfile):
        with open(sigfile, 'r', encoding='utf-8') as fh:
            signature = fh.read()
    else:
        signature = '%s <%s>' % (usercfg['name'], usercfg['email'])

outgoing = 0
    for jsondata in listing:
        slug_from = re.sub(r'\W', '_', jsondata['fromemail'])
        slug_subj = re.sub(r'\W', '_', jsondata['subject'])
        slug = '%s_%s' % (slug_from.lower(), slug_subj.lower())
        slug = re.sub(r'_+', '_', slug)
        jsondata['myname'] = usercfg['name']
        jsondata['myemail'] = usercfg['email']
        jsondata['signature'] = signature
        if 'pr_commit_id' in jsondata:
            # This is a pull request
            msg = generate_pr_thanks(gitdir, jsondata, branch)
        else:
            # This is a patch series
            msg = generate_am_thanks(gitdir, jsondata, branch, since)

if msg is None:
            continue

outgoing += 1
        outfile = os.path.join(outdir, '%s.thanks' % slug)
        logger.info('  Writing: %s', outfile)
        msg.set_charset('utf-8')
        msg.replace_header('Content-Transfer-Encoding', '8bit')
        with open(outfile, 'w') as fh:
            fh.write(msg.as_string(policy=b4.emlpolicy))
        logger.debug('Cleaning up: %s', jsondata['trackfile'])
        fullpath = os.path.join(datadir, jsondata['trackfile'])
        os.rename(fullpath, '%s.sent' % fullpath)
    logger.info('---')
    if not outgoing:
        logger.info('No thanks necessary.')
        return

logger.debug('Wrote %s thank-you letters', outgoing)
    logger.info('You can now run:')
    logger.info('  git send-email %s/*.thanks', outdir)

def list_tracked():
    # find all tracked bits
    tracked = list()
    datadir = b4.get_data_dir()
    paths = sorted(Path(datadir).iterdir(), key=os.path.getmtime)
    for fullpath in paths:
        if fullpath.suffix not in ('.pr', '.am'):
            continue
        with fullpath.open('r', encoding='utf-8') as fh:
            jsondata = json.load(fh)
            jsondata['trackfile'] = fullpath.name
            if fullpath.suffix == '.pr':
                jsondata['pr_commit_id'] = fullpath.stem
        tracked.append(jsondata)
    return tracked

def write_tracked(tracked):
    counter = 1
    config = b4.get_main_config()
    logger.info('Currently tracking:')
    for entry in tracked:
        logger.info('%3d: %s', counter, entry['subject'])
        logger.info('       From: %s <%s>', entry['fromname'], entry['fromemail'])
        logger.info('       Date: %s', entry['sentdate'])
        logger.info('       Link: %s', config['linkmask'] % entry['msgid'])
        counter += 1

def send_selected(cmdargs):
    tracked = list_tracked()
    if not len(tracked):
        logger.info('Nothing to do')
        sys.exit(0)

if cmdargs.send == 'all':
        listing = tracked
    else:
        listing = list()
        for num in b4.parse_int_range(cmdargs.send, upper=len(tracked)):
            try:
                index = int(num) - 1
                listing.append(tracked[index])
            except ValueError:
                logger.critical('Please provide the number of the message')
                logger.info('---')
                write_tracked(tracked)
                sys.exit(1)
            except IndexError:
                logger.critical('Invalid index: %s', num)
                logger.info('---')
                write_tracked(tracked)
                sys.exit(1)
    if not len(listing):
        logger.info('Nothing to do')
        sys.exit(0)

wantbranch = get_wanted_branch(cmdargs)
    send_messages(listing, cmdargs.gitdir, cmdargs.outdir, wantbranch, cmdargs.since)
    sys.exit(0)

def discard_selected(cmdargs):
    tracked = list_tracked()
    if not len(tracked):
        logger.info('Nothing to do')
        sys.exit(0)

if cmdargs.discard == 'all':
        listing = tracked
    else:
        listing = list()
        for num in b4.parse_int_range(cmdargs.discard, upper=len(tracked)):
            try:
                index = int(num) - 1
                listing.append(tracked[index])
            except ValueError:
                logger.critical('Please provide the number of the message')
                logger.info('---')
                write_tracked(tracked)
                sys.exit(1)
            except IndexError:
                logger.critical('Invalid index: %s', num)
                logger.info('---')
                write_tracked(tracked)
                sys.exit(1)

if not len(listing):
        logger.info('Nothing to do')
        sys.exit(0)

datadir = b4.get_data_dir()
    logger.info('Discarding %s messages', len(listing))
    for jsondata in listing:
        fullpath = os.path.join(datadir, jsondata['trackfile'])
        os.rename(fullpath, '%s.discarded' % fullpath)
        logger.info('  Discarded: %s', jsondata['subject'])

sys.exit(0)

def check_stale_thanks(outdir):
    if os.path.exists(outdir):
        for entry in Path(outdir).iterdir():
            if entry.suffix == '.thanks':
                logger.critical('ERROR: Found existing .thanks files in: %s', outdir)
                logger.critical('       Please send them first (or delete if already sent).')
                logger.critical('       Refusing to run to avoid potential confusion.')
                sys.exit(1)

def get_wanted_branch(cmdargs):
    global BRANCH_INFO
    gitdir = cmdargs.gitdir
    if not cmdargs.branch:
        # Find out our current branch
        gitargs = ['symbolic-ref', '-q', 'HEAD']
        ecode, out = b4.git_run_command(gitdir, gitargs)
        if ecode > 0:
            logger.critical('Not able to get current branch (git symbolic-ref HEAD)')
            sys.exit(1)
        wantbranch = re.sub(r'^refs/heads/', '', out.strip())
        logger.debug('will check branch=%s', wantbranch)
    else:
        # Make sure it's a real branch
        gitargs = ['branch', '--format=%(refname)', '--list', '--all', cmdargs.branch]
        lines = b4.git_get_command_lines(gitdir, gitargs)
        if not len(lines):
            logger.critical('Requested branch not found in git branch --list --all %s', cmdargs.branch)
            sys.exit(1)
        wantbranch = cmdargs.branch

return wantbranch

def get_branch_info(gitdir, branch):
    global BRANCH_INFO
    if BRANCH_INFO is not None:
        return BRANCH_INFO

BRANCH_INFO = dict()

remotecfg = b4.get_config_from_git('branch\\.%s\\.*' % branch)
    if remotecfg is None or 'remote' not in remotecfg:
        # Did not find a matching branch entry, so look at remotes
        gitargs = ['remote', 'show']
        lines = b4.git_get_command_lines(gitdir, gitargs)
        if not len(lines):
            # No remotes? Hmm...
            return BRANCH_INFO

remote = None
        for entry in lines:
            if branch.find(f'{entry}/') == 0:
                remote = entry
                break

if remote is None:
            # Not found any matching remotes
            return BRANCH_INFO

BRANCH_INFO['remote'] = remote
        BRANCH_INFO['branch'] = branch.replace(f'{remote}/', '')

else:
        BRANCH_INFO['remote'] = remotecfg['remote']
        if 'merge' in remotecfg:
            BRANCH_INFO['branch'] = re.sub(r'^refs/heads/', '', remotecfg['merge'])

# Grab template overrides
    remotecfg = b4.get_config_from_git('remote\\.%s\\..*' % BRANCH_INFO['remote'])
    BRANCH_INFO.update(remotecfg)

return BRANCH_INFO

def main(cmdargs):
    usercfg = b4.get_user_config()
    if 'email' not in usercfg:
        logger.critical('Please set user.email in gitconfig to use this feature.')
        sys.exit(1)

if cmdargs.auto:
        check_stale_thanks(cmdargs.outdir)
        auto_thankanator(cmdargs)
    elif cmdargs.send:
        check_stale_thanks(cmdargs.outdir)
        send_selected(cmdargs)
    elif cmdargs.discard:
        discard_selected(cmdargs)
    else:
        tracked = list_tracked()
        if not len(tracked):
            logger.info('No thanks necessary.')
            sys.exit(0)
        write_tracked(tracked)
        logger.info('---')
        logger.info('You can send them using number ranges, e.g:')
        logger.info('  b4 ty -s 1-3,5,7-')
0707010000000C000041ED0000273B00000064000000025FEA205100000000000000000000000000000000000000000000001100000000b4-0.6.2+4/hooks0707010000000D000081ED0000273B00000064000000015FEA205100000199000000000000000000000000000000000000003500000000b4-0.6.2+4/hooks/sendemail-validate-attestation-hook#!/usr/bin/env bash
if which b4>/dev/null 2>&1; then
    # We have it in path, so just execute it
    b4 attest "${1}"
else
    # Assume we're symlinked into a b4 checkout
    REAL_SCRIPT=$(realpath -e ${BASH_SOURCE[0]})
    SCRIPT_TOP="${SCRIPT_TOP:-$(dirname ${REAL_SCRIPT})}"
    B4_TOP=$(realpath -e ${SCRIPT_TOP}/..)
    exec env PYTHONPATH="${B4_TOP}" python3 "${B4_TOP}/b4/command.py" attest "${1}"
fi
0707010000000E000041ED0000273B00000064000000025FEA205100000000000000000000000000000000000000000000000F00000000b4-0.6.2+4/man0707010000000F000081A40000273B00000064000000015FEA205100003267000000000000000000000000000000000000001400000000b4-0.6.2+4/man/b4.5.\" Man page generated from reStructuredText.
.
.TH B4 5 "2020-11-20" "0.6.0" ""
.SH NAME
B4 \- Work with code submissions in a public-inbox archive
.
.nr rst2man-indent-level 0
.
.de1 rstReportMargin
\\$1 \\n[an-margin]
level \\n[rst2man-indent-level]
level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
-
\\n[rst2man-indent0]
\\n[rst2man-indent1]
\\n[rst2man-indent2]
..
.de1 INDENT
.\" .rstReportMargin pre:
. RS \\$1
. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
. nr rst2man-indent-level +1
.\" .rstReportMargin post:
..
.de UNINDENT
. RE
.\" indent \\n[an-margin]
.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
.nr rst2man-indent-level -1
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.SH SYNOPSIS
.sp
b4 {mbox,am,attest,pr,ty,diff} [options]
.SH DESCRIPTION
.sp
This is a helper utility to work with patches and pull requests made
available via a public\-inbox archive like lore.kernel.org. It is
written to make it easier to participate in a patch\-based workflows,
like those used in the Linux kernel development.
.sp
The name "b4" was chosen for ease of typing and because B\-4 was the
precursor to Lore and Data in the Star Trek universe.
.SH SUBCOMMANDS
.INDENT 0.0
.IP \(bu 2
\fIb4 mbox\fP: Download a thread as an mbox file
.IP \(bu 2
\fIb4 am\fP: Create an mbox file that is ready to git\-am
.IP \(bu 2
\fIb4 pr\fP: Work with pull requests
.IP \(bu 2
\fIb4 diff\fP: Show range\-diff style diffs between patch versions
.IP \(bu 2
\fIb4 ty\fP: (EXPERIMENTAL) Create templated replies for processed patches and pull requests
.IP \(bu 2
\fIb4 attest\fP: (EXPERIMENTAL) Add cryptographic attestation to patches
.UNINDENT
.SH OPTIONS
.INDENT 0.0
.TP
.B \-h\fP,\fB  \-\-help
show this help message and exit
.TP
.B \-d\fP,\fB  \-\-debug
Add more debugging info to the output (default: False)
.TP
.B \-q\fP,\fB  \-\-quiet
Output critical information only (default: False)
.UNINDENT
.SH SUBCOMMAND OPTIONS
.SS b4 mbox
.INDENT 0.0
.TP
.B usage:
b4 mbox [\-h] [\-o OUTDIR] [\-p USEPROJECT] [\-c] [\-n WANTNAME] [\-m LOCALMBOX] [msgid]
.TP
.B positional arguments:
msgid                 Message ID to process, or pipe a raw message
.TP
.B optional arguments:
.INDENT 7.0
.TP
.B \-h\fP,\fB  \-\-help
show this help message and exit
.TP
.BI \-o \ OUTDIR\fP,\fB \ \-\-outdir \ OUTDIR
Output into this directory (or use \- to output mailbox contents to stdout)
.TP
.BI \-p \ USEPROJECT\fP,\fB \ \-\-use\-project \ USEPROJECT
Use a specific project instead of guessing (linux\-mm, linux\-hardening, etc)
.TP
.B \-c\fP,\fB  \-\-check\-newer\-revisions
Check if newer patch revisions exist
.TP
.BI \-n \ WANTNAME\fP,\fB \ \-\-mbox\-name \ WANTNAME
Filename to name the mbox file
.TP
.BI \-m \ LOCALMBOX\fP,\fB \ \-\-use\-local\-mbox \ LOCALMBOX
Instead of grabbing a thread from lore, process this mbox file
.TP
.B \-C\fP,\fB  \-\-no\-cache
Do not use local cache
.TP
.B \-f\fP,\fB  \-\-filter\-dupes
When adding messages to existing maildir, filter out duplicates
.UNINDENT
.UNINDENT
.sp
\fIExample\fP: b4 mbox \fI\%20200313231252.64999\-1\-keescook@chromium.org\fP
.SS b4 am
.INDENT 0.0
.TP
.B usage:
b4 am [\-h] [\-o OUTDIR] [\-p USEPROJECT] [\-c] [\-n WANTNAME] [\-m LOCALMBOX] [\-v WANTVER] [\-t] [\-T] [\-s] [\-l] [\-Q] [msgid]
.TP
.B positional arguments:
msgid                 Message ID to process, or pipe a raw message
.TP
.B optional arguments:
.INDENT 7.0
.TP
.B \-h\fP,\fB  \-\-help
show this help message and exit
.TP
.BI \-o \ OUTDIR\fP,\fB \ \-\-outdir \ OUTDIR
Output into this directory (or use \- to output mailbox contents to stdout)
.TP
.BI \-p \ USEPROJECT\fP,\fB \ \-\-use\-project \ USEPROJECT
Use a specific project instead of guessing (linux\-mm, linux\-hardening, etc)
.TP
.B \-c\fP,\fB  \-\-check\-newer\-revisions
Check if newer patch revisions exist
.TP
.BI \-n \ WANTNAME\fP,\fB \ \-\-mbox\-name \ WANTNAME
Filename to name the mbox file
.TP
.BI \-m \ LOCALMBOX\fP,\fB \ \-\-use\-local\-mbox \ LOCALMBOX
Instead of grabbing a thread from lore, process this mbox file
.TP
.B \-C\fP,\fB  \-\-no\-cache
Do not use local cache
.TP
.BI \-v \ WANTVER\fP,\fB \ \-\-use\-version \ WANTVER
Get a specific version of the patch/series
.TP
.B \-t\fP,\fB  \-\-apply\-cover\-trailers
Apply trailers sent to the cover letter to all patches
.TP
.B \-S\fP,\fB  \-\-sloppy\-trailers
Apply trailers without email address match checking
.TP
.B \-T\fP,\fB  \-\-no\-add\-trailers
Do not add or sort any trailers
.TP
.B \-s\fP,\fB  \-\-add\-my\-sob
Add your own signed\-off\-by to every patch
.TP
.B \-l\fP,\fB  \-\-add\-link
Add a lore.kernel.org/r/ link to every patch
.TP
.B \-Q\fP,\fB  \-\-quilt\-ready
Save mbox patches in a quilt\-ready folder
.TP
.BI \-P \ CHERRYPICK\fP,\fB \ \-\-cherry\-pick \ CHERRYPICK
Cherry\-pick a subset of patches (e.g. "\-P 1\-2,4,6\-", "\-P _" to use just the msgid specified, or "\-P *globbing*" to match on commit subject)
.TP
.B \-g\fP,\fB  \-\-guess\-base
Try to guess the base of the series (if not specified)
.TP
.B \-3\fP,\fB  \-\-prep\-3way
Prepare for a 3\-way merge (tries to ensure that all index blobs exist by making a fake commit range)
.TP
.B \-\-cc\-trailers
Copy all Cc\(aqd addresses into Cc: trailers, if not already present
.TP
.B \-\-no\-cover
Do not save the cover letter (on by default when using \-o \-)
.UNINDENT
.UNINDENT
.sp
\fIExample\fP: b4 am \fI\%20200313231252.64999\-1\-keescook@chromium.org\fP
.SS b4 attest
.sp
usage: b4 attest [\-h] [\-f SENDER] [\-n] [\-o OUTPUT] patchfile [patchfile ...]
.INDENT 0.0
.TP
.B positional arguments:
patchfile             Patches to attest
.TP
.B optional arguments:
.INDENT 7.0
.TP
.B \-h\fP,\fB  \-\-help
show this help message and exit
.TP
.BI \-f \ SENDER\fP,\fB \ \-\-from \ SENDER
OBSOLETE: this option does nothing and will be removed
.TP
.B \-n\fP,\fB  \-\-no\-submit
OBSOLETE: this option does nothing and will be removed
.TP
.BI \-o \ OUTPUT\fP,\fB \ \-\-output \ OUTPUT
OBSOLETE: this option does nothing and will be removed
.UNINDENT
.UNINDENT
.sp
\fIExample\fP: b4 attest output/*.patch
.SS b4 pr
.INDENT 0.0
.TP
.B usage:
command.py pr [\-h] [\-g GITDIR] [\-b BRANCH] [\-c] [\-e] [\-o OUTMBOX] [msgid]
.TP
.B positional arguments:
msgid                 Message ID to process, or pipe a raw message
.TP
.B optional arguments:
.INDENT 7.0
.TP
.B \-h\fP,\fB  \-\-help
show this help message and exit
.TP
.BI \-g \ GITDIR\fP,\fB \ \-\-gitdir \ GITDIR
Operate on this git tree instead of current dir
.TP
.BI \-b \ BRANCH\fP,\fB \ \-\-branch \ BRANCH
Check out FETCH_HEAD into this branch after fetching
.TP
.B \-c\fP,\fB  \-\-check
Check if pull request has already been applied
.TP
.B \-e\fP,\fB  \-\-explode
Convert a pull request into an mbox full of patches
.TP
.BI \-o \ OUTMBOX\fP,\fB \ \-\-output\-mbox \ OUTMBOX
Save exploded messages into this mailbox (default: msgid.mbx)
.UNINDENT
.UNINDENT
.sp
\fIExample\fP: b4 pr \fI\%202003292120.2BDCB41@keescook\fP
.SS b4 ty
.INDENT 0.0
.TP
.B usage:
b4 ty [\-h] [\-g GITDIR] [\-o OUTDIR] [\-l] [\-s SEND [SEND ...]] [\-d DISCARD [DISCARD ...]] [\-a] [\-b BRANCH] [\-\-since SINCE]
.TP
.B optional arguments:
.INDENT 7.0
.TP
.B \-h\fP,\fB  \-\-help
show this help message and exit
.TP
.BI \-g \ GITDIR\fP,\fB \ \-\-gitdir \ GITDIR
Operate on this git tree instead of current dir
.TP
.BI \-o \ OUTDIR\fP,\fB \ \-\-outdir \ OUTDIR
Write thanks files into this dir (default=.)
.TP
.B \-l\fP,\fB  \-\-list
List pull requests and patch series you have retrieved
.TP
.BI \-s \ SEND\fP,\fB \ \-\-send \ SEND
Generate thankyous for specific entries from \-l (e.g.: 1,3\-5,7\-; or "all")
.TP
.BI \-d \ DISCARD\fP,\fB \ \-\-discard \ DISCARD
Discard specific messages from \-l (e.g.: 1,3\-5,7\-; or "all")
.TP
.B \-a\fP,\fB  \-\-auto
Use the Auto\-Thankanator to figure out what got applied/merged
.TP
.BI \-b \ BRANCH\fP,\fB \ \-\-branch \ BRANCH
The branch to check against, instead of current
.TP
.BI \-\-since \ SINCE
The \-\-since option to use when auto\-matching patches (default=1.week)
.UNINDENT
.UNINDENT
.sp
\fIExample\fP: b4 ty \-\-auto
.SS b4 diff
.sp
usage: b4 diff [\-h] [\-g GITDIR] [\-p USEPROJECT] [\-C] [\-v WANTVERS [WANTVERS ...]] [\-n] [\-o OUTDIFF] [\-c] [\-m AMBOX AMBOX] [msgid]
.INDENT 0.0
.TP
.B positional arguments:
msgid                 Message ID to process, pipe a raw message, or use \-m
.UNINDENT
.sp
optional arguments:
.INDENT 0.0
.INDENT 3.5
.INDENT 0.0
.TP
.B \-h\fP,\fB  \-\-help
show this help message and exit
.TP
.BI \-g \ GITDIR\fP,\fB \ \-\-gitdir \ GITDIR
Operate on this git tree instead of current dir
.TP
.BI \-p \ USEPROJECT\fP,\fB \ \-\-use\-project \ USEPROJECT
Use a specific project instead of guessing (linux\-mm, linux\-hardening, etc)
.TP
.B \-C\fP,\fB  \-\-no\-cache
Do not use local cache
.UNINDENT
.INDENT 0.0
.TP
.B \-v WANTVERS [WANTVERS ...], \-\-compare\-versions WANTVERS [WANTVERS ...]
Compare specific versions instead of latest and one before that, e.g. \-v 3 5
.UNINDENT
.INDENT 0.0
.TP
.B \-n\fP,\fB  \-\-no\-diff
Do not generate a diff, just show the command to do it
.TP
.BI \-o \ OUTDIFF\fP,\fB \ \-\-output\-diff \ OUTDIFF
Save diff into this file instead of outputting to stdout
.TP
.B \-c\fP,\fB  \-\-color
Force color output even when writing to file
.UNINDENT
.INDENT 0.0
.TP
.B \-m AMBOX AMBOX, \-\-compare\-am\-mboxes AMBOX AMBOX
Compare two mbx files prepared with "b4 am"
.UNINDENT
.UNINDENT
.UNINDENT
.sp
\fIExample\fP: b4 diff \fI\%20200526205322.23465\-1\-mic@digikod.net\fP
.SH CONFIGURATION
.sp
B4 configuration is handled via git\-config(1), so you can store it in
either the toplevel $HOME/.gitconfig file, or in a per\-repository
.git/config file if your workflow changes per project.
.sp
Default configuration, with explanations:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
[b4]
   # Where to look up threads by message id
   midmask = https://lore.kernel.org/r/%s
   #
   # When recording Link: trailers, use this mask
   linkmask = https://lore.kernel.org/r/%s
   #
   # When processing thread trailers, sort them in this order.
   # Can use shell\-globbing and must end with ,*
   # Some sorting orders:
   #trailer\-order=link*,fixes*,cc*,reported*,suggested*,original*,co\-*,tested*,reviewed*,acked*,signed\-off*,*
   #trailer\-order = fixes*,reported*,suggested*,original*,co\-*,signed\-off*,tested*,reviewed*,acked*,cc*,link*,*
   trailer\-order = _preserve_
   #
   # Attestation\-checking configuration parameters
   # off: do not bother checking attestation
   # check: print an attaboy when attestation is found
   # softfail: print a warning when no attestation found
   # hardfail: exit with an error when no attestation found
   attestation\-policy = check
   #
   # Fall back to checking DKIM header if we don\(aqt find any other
   # attestations present?
   attestation\-check\-dkim = yes
   #
   # "gpg" (whatever gpg is configured to do) or "tofu" to force TOFU mode
   # If you don\(aqt already have a carefully maintained web of trust setup, it is
   # strongly recommended to set this to "tofu"
   attestation\-trust\-model = gpg
   #
   # How strict should we be when comparing the email address in From to the
   # email addresses in the key\(aqs UIDs?
   # strict: must match one of the uids on the key to pass
   # loose: any valid and trusted key will be accepted
   attestation\-uid\-match = loose
   #
   # When showing attestation check results, do you like "fancy" (color, unicode)
   # or simple checkmarks?
   attestation\-checkmarks = fancy
   #
   # How long before we consider attestation to be too old?
   attestation\-staleness\-days = 30
   #
   # You can point this at a non\-default home dir, if you like, or leave out to
   # use the OS default.
   attestation\-gnupghome = None
   #
   # If this is not set, we\(aqll use what we find in
   # git\-config for gpg.program; and if that\(aqs not set,
   # we\(aqll use "gpg" and hope for the best
   gpgbin = None
   #
   # How long to keep downloaded threads in cache (minutes)?
   cache\-expire = 10
   # Used when creating summaries for b4 ty, and can be set to a value like
   # thanks\-commit\-url\-mask = https://git.kernel.org/username/c/%.12s
   # See this page for more info on convenient git.kernel.org shorterners:
   # https://korg.wiki.kernel.org/userdoc/git\-url\-shorterners
   thanks\-commit\-url\-mask = None
   # See thanks\-pr\-template.example. If not set, a default template will be used.
   thanks\-pr\-template = None
   # See thanks\-am\-template.example. If not set, a default template will be used.
   thanks\-am\-template = None
.ft P
.fi
.UNINDENT
.UNINDENT
.SH SUPPORT
.sp
Please email \fI\%tools@linux.kernel.org\fP with support requests,
or browse the list archive at \fI\%https://linux.kernel.org/g/tools\fP\&.
.SH AUTHOR
mricon@kernel.org

License: GPLv2+
.SH COPYRIGHT
The Linux Foundation and contributors
.\" Generated by docutils manpage writer.
.
07070100000010000081A40000273B00000064000000015FEA205100002DB0000000000000000000000000000000000000001800000000b4-0.6.2+4/man/b4.5.rstB4
==
----------------------------------------------------
Work with code submissions in a public-inbox archive
----------------------------------------------------

:Author:    mricon@kernel.org
:Date:      2020-11-20
:Copyright: The Linux Foundation and contributors
:License:   GPLv2+
:Version:   0.6.0
:Manual section: 5

SYNOPSIS
--------
b4 {mbox,am,attest,pr,ty,diff} [options]

DESCRIPTION
-----------
This is a helper utility to work with patches and pull requests made
available via a public-inbox archive like lore.kernel.org. It is
written to make it easier to participate in a patch-based workflows,
like those used in the Linux kernel development.

The name "b4" was chosen for ease of typing and because B-4 was the
precursor to Lore and Data in the Star Trek universe.

SUBCOMMANDS
-----------
* *b4 mbox*: Download a thread as an mbox file
* *b4 am*: Create an mbox file that is ready to git-am
* *b4 pr*: Work with pull requests
* *b4 diff*: Show range-diff style diffs between patch versions
* *b4 ty*: (EXPERIMENTAL) Create templated replies for processed patches and pull requests
* *b4 attest*: (EXPERIMENTAL) Add cryptographic attestation to patches

OPTIONS
-------
-h, --help            show this help message and exit
-d, --debug           Add more debugging info to the output (default: False)
-q, --quiet           Output critical information only (default: False)

SUBCOMMAND OPTIONS
------------------
b4 mbox
~~~~~~~
usage:
  b4 mbox [-h] [-o OUTDIR] [-p USEPROJECT] [-c] [-n WANTNAME] [-m LOCALMBOX] [msgid]

positional arguments:
  msgid                 Message ID to process, or pipe a raw message

*Example*: b4 mbox 20200313231252.64999-1-keescook@chromium.org

b4 am
~~~~~
usage:
  b4 am [-h] [-o OUTDIR] [-p USEPROJECT] [-c] [-n WANTNAME] [-m LOCALMBOX] [-v WANTVER] [-t] [-T] [-s] [-l] [-Q] [msgid]

positional arguments:
  msgid                 Message ID to process, or pipe a raw message

optional arguments:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        Output into this directory (or use - to output mailbox contents to stdout)
  -p USEPROJECT, --use-project USEPROJECT
                        Use a specific project instead of guessing (linux-mm, linux-hardening, etc)
  -c, --check-newer-revisions
                        Check if newer patch revisions exist
  -n WANTNAME, --mbox-name WANTNAME
                        Filename to name the mbox file
  -m LOCALMBOX, --use-local-mbox LOCALMBOX
                        Instead of grabbing a thread from lore, process this mbox file
  -C, --no-cache        Do not use local cache
  -v WANTVER, --use-version WANTVER
                        Get a specific version of the patch/series
  -t, --apply-cover-trailers
                        Apply trailers sent to the cover letter to all patches
  -S, --sloppy-trailers
                        Apply trailers without email address match checking
  -T, --no-add-trailers
                        Do not add or sort any trailers
  -s, --add-my-sob      Add your own signed-off-by to every patch
  -l, --add-link        Add a lore.kernel.org/r/ link to every patch
  -Q, --quilt-ready     Save mbox patches in a quilt-ready folder
  -P CHERRYPICK, --cherry-pick CHERRYPICK
                        Cherry-pick a subset of patches (e.g. "-P 1-2,4,6-", "-P _" to use just the msgid specified, or "-P \*globbing\*" to match on commit subject)
  -g, --guess-base
                        Try to guess the base of the series (if not specified)
  -3, --prep-3way
                        Prepare for a 3-way merge (tries to ensure that all index blobs exist by making a fake commit range)
  --cc-trailers
                        Copy all Cc'd addresses into Cc: trailers, if not already present
  --no-cover
                        Do not save the cover letter (on by default when using -o -)

*Example*: b4 am 20200313231252.64999-1-keescook@chromium.org

b4 attest
~~~~~~~~~
usage: b4 attest [-h] [-f SENDER] [-n] [-o OUTPUT] patchfile [patchfile ...]

positional arguments:
  patchfile             Patches to attest

optional arguments:
  -h, --help            show this help message and exit
  -f SENDER, --from SENDER
                        OBSOLETE: this option does nothing and will be removed
  -n, --no-submit       OBSOLETE: this option does nothing and will be removed
  -o OUTPUT, --output OUTPUT
                        OBSOLETE: this option does nothing and will be removed
 
*Example*: b4 attest output/\*.patch

b4 pr
~~~~~
usage:
  command.py pr [-h] [-g GITDIR] [-b BRANCH] [-c] [-e] [-o OUTMBOX] [msgid]

positional arguments:
  msgid                 Message ID to process, or pipe a raw message

optional arguments:
  -h, --help            show this help message and exit
  -g GITDIR, --gitdir GITDIR
                        Operate on this git tree instead of current dir
  -b BRANCH, --branch BRANCH
                        Check out FETCH_HEAD into this branch after fetching
  -c, --check           Check if pull request has already been applied
  -e, --explode         Convert a pull request into an mbox full of patches
  -o OUTMBOX, --output-mbox OUTMBOX
                        Save exploded messages into this mailbox (default: msgid.mbx)

*Example*: b4 pr 202003292120.2BDCB41@keescook

b4 ty
~~~~~
usage:
  b4 ty [-h] [-g GITDIR] [-o OUTDIR] [-l] [-s SEND [SEND ...]] [-d DISCARD [DISCARD ...]] [-a] [-b BRANCH] [--since SINCE]

optional arguments:
  -h, --help            show this help message and exit
  -g GITDIR, --gitdir GITDIR
                        Operate on this git tree instead of current dir
  -o OUTDIR, --outdir OUTDIR
                        Write thanks files into this dir (default=.)
  -l, --list            List pull requests and patch series you have retrieved
  -s SEND, --send SEND  Generate thankyous for specific entries from -l (e.g.: 1,3-5,7-; or "all")
  -d DISCARD, --discard DISCARD
                        Discard specific messages from -l (e.g.: 1,3-5,7-; or "all")
  -a, --auto            Use the Auto-Thankanator to figure out what got applied/merged
  -b BRANCH, --branch BRANCH
                        The branch to check against, instead of current
  --since SINCE         The --since option to use when auto-matching patches (default=1.week)

*Example*: b4 ty --auto

b4 diff
~~~~~~~
usage: b4 diff [-h] [-g GITDIR] [-p USEPROJECT] [-C] [-v WANTVERS [WANTVERS ...]] [-n] [-o OUTDIFF] [-c] [-m AMBOX AMBOX] [msgid]

positional arguments:
  msgid                 Message ID to process, pipe a raw message, or use -m

optional arguments:

-h, --help            show this help message and exit
  -g GITDIR, --gitdir GITDIR
                        Operate on this git tree instead of current dir
  -p USEPROJECT, --use-project USEPROJECT
                        Use a specific project instead of guessing (linux-mm, linux-hardening, etc)
  -C, --no-cache        Do not use local cache

-v WANTVERS [WANTVERS ...], --compare-versions WANTVERS [WANTVERS ...]
                        Compare specific versions instead of latest and one before that, e.g. -v 3 5

-n, --no-diff
                        Do not generate a diff, just show the command to do it

-o OUTDIFF, --output-diff OUTDIFF
                        Save diff into this file instead of outputting to stdout
  -c, --color
                        Force color output even when writing to file

-m AMBOX AMBOX, --compare-am-mboxes AMBOX AMBOX
                        Compare two mbx files prepared with "b4 am"

*Example*: b4 diff 20200526205322.23465-1-mic@digikod.net

CONFIGURATION
-------------
B4 configuration is handled via git-config(1), so you can store it in
either the toplevel $HOME/.gitconfig file, or in a per-repository
.git/config file if your workflow changes per project.

Default configuration, with explanations::

[b4]
      # Where to look up threads by message id
      midmask = https://lore.kernel.org/r/%s
      #
      # When recording Link: trailers, use this mask
      linkmask = https://lore.kernel.org/r/%s
      #
      # When processing thread trailers, sort them in this order.
      # Can use shell-globbing and must end with ,*
      # Some sorting orders:
      #trailer-order=link*,fixes*,cc*,reported*,suggested*,original*,co-*,tested*,reviewed*,acked*,signed-off*,*
      #trailer-order = fixes*,reported*,suggested*,original*,co-*,signed-off*,tested*,reviewed*,acked*,cc*,link*,*
      trailer-order = _preserve_
      #
      # Attestation-checking configuration parameters
      # off: do not bother checking attestation
      # check: print an attaboy when attestation is found
      # softfail: print a warning when no attestation found
      # hardfail: exit with an error when no attestation found
      attestation-policy = check
      #
      # Fall back to checking DKIM header if we don't find any other
      # attestations present?
      attestation-check-dkim = yes
      #
      # "gpg" (whatever gpg is configured to do) or "tofu" to force TOFU mode
      # If you don't already have a carefully maintained web of trust setup, it is
      # strongly recommended to set this to "tofu"
      attestation-trust-model = gpg
      #
      # How strict should we be when comparing the email address in From to the
      # email addresses in the key's UIDs?
      # strict: must match one of the uids on the key to pass
      # loose: any valid and trusted key will be accepted
      attestation-uid-match = loose
      #
      # When showing attestation check results, do you like "fancy" (color, unicode)
      # or simple checkmarks?
      attestation-checkmarks = fancy
      #
      # How long before we consider attestation to be too old?
      attestation-staleness-days = 30
      #
      # You can point this at a non-default home dir, if you like, or leave out to
      # use the OS default.
      attestation-gnupghome = None
      #
      # If this is not set, we'll use what we find in
      # git-config for gpg.program; and if that's not set,
      # we'll use "gpg" and hope for the best
      gpgbin = None
      #
      # How long to keep downloaded threads in cache (minutes)?
      cache-expire = 10
      # Used when creating summaries for b4 ty, and can be set to a value like
      # thanks-commit-url-mask = https://git.kernel.org/username/c/%.12s
      # See this page for more info on convenient git.kernel.org shorterners:
      # https://korg.wiki.kernel.org/userdoc/git-url-shorterners
      thanks-commit-url-mask = None
      # See thanks-pr-template.example. If not set, a default template will be used.
      thanks-pr-template = None
      # See thanks-am-template.example. If not set, a default template will be used.
      thanks-am-template = None

SUPPORT
-------
Please email tools@linux.kernel.org with support requests,
or browse the list archive at https://linux.kernel.org/g/tools.
07070100000011000081A40000273B00000064000000015FEA2051000000D8000000000000000000000000000000000000001C00000000b4-0.6.2+4/requirements.txtrequests ~= 2.24.0
# These are optional, needed for attestation features
dnspython~=2.0.0
dkimpy~=1.0.5
# These may be required in the future for other patch attestation features
#pycryptodomex~=3.9.9
#PyNaCl~=1.4.0
07070100000012000081A40000273B00000064000000015FEA2051000005EA000000000000000000000000000000000000001400000000b4-0.6.2+4/setup.py#!/usr/bin/env python3

import os
import re
from setuptools import setup

# Utility function to read the README file.
# Used for the long_description.  It's nice, because now 1) we have a top level
# README file and 2) it's easier to type in the README file than to put a raw
# string in below ...

def read(fname):
    return open(os.path.join(os.path.dirname(__file__), fname)).read()

def find_version(source):
    version_file = read(source)
    version_match = re.search(r"^__VERSION__ = ['\"]([^'\"]*)['\"]", version_file, re.M)
    if version_match:
        return version_match.group(1)
    raise RuntimeError("Unable to find version string.")

NAME = 'b4'

setup(
    version=find_version('b4/__init__.py'),
    url='https://git.kernel.org/pub/scm/utils/b4/b4.git/tree/README.rst',
    project_urls={
        'Community': 'https://linux.kernel.org/g/tools'
    },
    name=NAME,
    description='A tool to work with public-inbox and patch archives',
    author='Konstantin Ryabitsev',
    author_email='mricon@kernel.org',
    packages=['b4'],
    license='GPLv2+',
    long_description=read('man/b4.5.rst'),
    long_description_content_type='text/x-rst',
    data_files = [('share/man/man5', ['man/b4.5'])],
    keywords=['git', 'lore.kernel.org', 'patches'],
    install_requires=[
        'requests~=2.24.0',
        'dkimpy~=1.0.5',
        'dnspython~=2.0.0',
    ],
    python_requires='>=3.6',
    entry_points={
        'console_scripts': [
            'b4=b4.command:cmd'
        ],
    },
)
07070100000013000081A40000273B00000064000000015FEA205100000595000000000000000000000000000000000000002600000000b4-0.6.2+4/thanks-am-template.example# Lines starting with '#' will be removed
# You can have two different templates for responding to
# pull requests and for responding to patch series, though
# in reality the difference will probably be one word:
# "merged/pulled" vs. "applied".
# Keeping it short and sweet is preferred.
#
On ${sentdate}, ${fromname} wrote:
# quote will be limited to 5-6 lines, respecting paragraphs
${quote}

# You can also use ${branch} and ${treename} if you set
# b4.thanks-treename in your config, e.g.:
#Applied to ${treename} (${branch}), thanks!
#
# If you track multiple remotes in the same repo, then you can add
# the following values to [remote], to be loaded when you run
# b4 ty -b foo/branchname:
# [remote "foo"]
#   url = https://...
#   fetch = ...
#   b4-treename = uname/sound.git
#   b4-commit-url-mask = https://git.kernel.org/uname/sound/c/%.8s
Applied to ${branch}, thanks!

# for patch series, the summary is a list of each patch with a link
# to the commit id in your tree, so you probably want to set
# b4.thanks-commit-url-mask in gitconfig to a value like:
# [b4]
#   thanks-commit-url-mask = https://git.kernel.org/username/c/%.12s
#
# Check this page for info on convenient URL shorteners:
# https://korg.wiki.kernel.org/userdoc/git-url-shorterners
${summary}

Best regards,
-- 
# if ~/.signature exists, it will be put here, otherwise
# the contents will be "user.name <user.email>" from gitconfig
${signature}
07070100000014000081A40000273B00000064000000015FEA20510000057B000000000000000000000000000000000000002600000000b4-0.6.2+4/thanks-pr-template.example# Lines starting with '#' will be removed
# You can have two different templates for responding to
# pull requests and for responding to patch series, though
# in reality the difference will probably be one word:
# "merged/pulled" vs. "applied".
# Keeping it short and sweet is preferred.
#
On ${sentdate}, ${fromname} wrote:
# quote will be limited to 5-6 lines, respecting paragraphs
${quote}

# You can also use ${branch} and ${treename} if you set
# b4.thanks-treename in your config, e.g.:
#Merged into ${treename} (${branch}), thanks!
#
# If you track multiple remotes in the same repo, then you can add
# the following values to [remote], to be loaded when you run
# b4 ty -b foo/branchname:
# [remote "foo"]
#   url = https://...
#   fetch = ...
#   b4-treename = uname/sound.git
#   b4-commit-url-mask = https://git.kernel.org/uname/sound/c/%.8s
Merged into ${branch}, thanks!

# for pull requests, the summary is a one-liner with the merge commit,
# so you probably want to set b4.thanks-commit-url-mask in gitconfig
# to a value like:
# [b4]
#   thanks-commit-url-mask = https://git.kernel.org/username/c/%.12s
#
# Check this page for info on convenient URL shorteners:
# https://korg.wiki.kernel.org/userdoc/git-url-shorterners
${summary}

Best regards,
-- 
# if ~/.signature exists, it will be put here, otherwise
# the contents will be "user.name <user.email>" from gitconfig
${signature}
07070100000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000B00000000TRAILER!!!451 blocks

Places

File b4-0.6.2+4.obscpio of Package b4 (Revision 0b9d4f36093b893afcbad0c766bbc24d)

Places