![[LWN Logo]](/images/lwn.banner.gif)
From: Jeff Graham <demit@best.com>
Subject: FAQ updated, New address for faq info
To: security-audit@ferret.lmh.ox.ac.uk (security audit)
Date: Mon, 20 Mar 2000 17:05:55 -0800 (PST)
Please note the new address for FAQ submissions lsap@demit.net
Thanks
Archive-name: security-audit-faq
Security-Audit's Frequently Asked Questions (released)
$Id: audit-faq,v 1.9 2000/03/21 01:01:08 demit Exp $
This is a collection of common questions posted to the security-audit
mailing list once a month. It is intended to reduce the traffic to this
list by providing quick answers to common questions.
This is a blatant rip off of the format of the sun-managers FAQ which
itself credits a similar FAQ for comp.windows.x (can we say rfc for faqs
maybe?), questions marked with a '+' indicate questions new to this
version; those with significant changes of content since the last
version are marked by '*'; those with minor changes of content that may
be of interest are marked by '-'.
The current FAQ maintainer is Jeff Graham <lsap@demit.net>. All
corrections, submissions and FAQ administration-related messages should
go to this address (for now.)
Additional content and HTML conversion added by Greg Patterson
<gomer@gomer.mlink.net>
------------------------------------------------------------------------
Questions
1. The Security-Audit Mailing list.
1.1) How do I read, join, post to, or remove myself from the
security-audit mailing list?
1.2) What is the purpose of this list?
1.3) Are there any archives for the security-audit list?
1.4) What should I post to this list?
* 1.5) What other security resources are available?
1.6) What sort of thing is considered "off topic" for the list?
* 1.7) What programs have been "fixed" by the audit?
2. Bugs and what to do about them.
2.1) I have found a bug, what should I do?
2.2) I reported a bug to the programmer(s) and never got an
answer, what should I do now?
2.3) I think this is a bug, but I am not sure. What should I
do?
2.4) What kind of things should I be looking for?
* 2.5) Are there any tools that can help me check for bugs?
2.6) Where are some secure programming references?
3. Auditing tasks.
3.1) How do I suggest something I feel needs auditing but do
not have time to do myself?
3.2) How can I find something that needs auditing and is not
currently being worked on by anyone?
4. Other security information.
4.1) What about encrypted file systems?
* 4.2) What are some automatic tools I can use to improve
security?
-------------------------------------------------------------------------
Answers
-------------------------------------------------------------------------
-------------------------------------------------------------------------
1. The Security-Audit Mailing list
-------------------------------------------------------------------------
Subject: 1.1) How do I read, join, post to, or remove myself from the
security-audit mailing list.
To read this mailing list you must be subscribed as mentioned below.
To subscribe to this mailing list you should send an empty note to
<security-audit-subscribe@ferret.lmh.ox.ac.uk>. Once you send a mail to
this address, you will receive a mail back as confirmation that you wish
to subscribe, if your mail program understands Reply-To headers, all you
have to do is hit reply (you don't even need to send the body of it
back). Once this is done, you are subscribed.
To subscribe other addresses send mail to
<security-audit-subscribe-USER=host.domain@ferret.lmh.ox.ac.uk>, so for
instance to subscribe mylogin@myhost.com you would send to
<security-audit-subscribe-mylogin=myhost.com@ferret.lmh.ox.ac.uk>. When
subscribing other addresses than the one you are mailing from, the
"other" address will receive a confirmation message which will need to
be replied to.
To post to the Security-audit mailing list you just send mail to the
<security-audit@ferret.lmh.ox.ac.uk> address. Anything you send there
will be sent to the list.
To unsubscribe from this mailing list, send an empty note to
<security-audit-unsubscribe@ferret.lmh.ox.ac.uk>.
There is not a digest form of security-audit at this time.
The latest version of the FAQ (this file) will be posted to the list
once a month and will also be available at:
http://ferret.lmh.ox.ac.uk/~security
An HTML version of this file will be available on:
http://www.linuxhelp.org/lsap.shtml
-------------------------------------------------------------------------
Subject: 1.2) What is the purpose of this list?
This list has been formed to coordinate a group of people who are
attempting to make the linux operating system more secure. This is
being done by reviewing commonly run code for security issues. This
group of people is all volunteers and is doing this project on their own
time (for now.) The group is attempting to make a sort of "verified"
linux. This is a massive task but every fix that is put in place is one
less exploit waiting to happen.
-------------------------------------------------------------------------
Subject: 1.3) Are there any archives for the security-audit list?
There is an unofficial archive of the mailing list at:
http://lists.nas.nasa.gov/archives/ext/linux-security-audit/
It was started 2 days after the creating of the mailing list.
The maintainer is: Tyler Allison <allison@mail.arc.nasa.gov>
There is a site (maintained by Alan Cox) that has many of the fixes
discussed in RedHat RPM format at
ftp://ftp.uk.linux.org/pub/linux/alan/Security.
-------------------------------------------------------------------------
Subject: 1.4) What should I post to this list?
The list is currently open so you can post anything you wish to the
list. However, unless you wish to start a flame war against yourself
you may wish to stay at least marginally on topic. Typically things
that should be posted to the list included: bugs found, requests for
information on questionable code, packages/programs you are currently
auditing (including messages like "Is anyone already auditing XXX?",)
ideas for auditing techniques, and so on. Please try to keep your posts
succinct and try to keep the included amount of code to revelant
amounts.
-------------------------------------------------------------------------
Subject: 1.5) What other security resources are available?
Some resources:
FTP:
coast.cs.purdue.edu
ftp.cert.org
ftp.redhat.com
ftp.tux.org (/pub/dclug/jam or /pub/tux/jam)
ftp.linux.org.uk (/pub/linux/alan/Security NON OFFICIAL)
ftp.dataforce.net (/pub/solar)
ftp.porcupine.org (/pub/security)
HTTP:
http://www.l0pht.com/
http://www.sun.com/sunsolveonline (some secure programs
references)
http://olympus.cs.ucdavis.edu/~bishop/secprog.html
http://www.sun.com/sunworldonline/swol-04-1998/swol-04\
-security.html
http://www.pobox.com/~kragen/security-holes.html
http://www.whitefang.com/sup
http://www.rootshell.com/
http://packetstorm.securify.com
http://ferret.lmh.ox.ac.uk/~chris/rhbugs.txt (restarted for
redhat 5.2)
http://www.false.com/security
http://www.suse.de/~marc
http://www.cs.colorado.edu/homes/zorn/public_html\
/MallocDebug.html
http://SecurityPortal.com (misc. security information)
http://www.w00w00.org/articles.html (some how to exploit
resources)
http://www.nfr.com (network flight recorder)
http://info.cert.org/pub/cert_advisories
http://ciac.llnl.gov
http://www.cs.purdue.edu/coast
http://www.icsa.net/services/consortia/anti-virus/lab.html
http://www.auscert.org.au
http://spam.abuse.net/spam/tools/mailblock.html#filters
http://www.net-dev.com/ned-03-1998/ned-03-security.html
http://www.cs.princeton.edu/sip
http://www.iss.net/xforce
http://www.squirrel.com/squirrel/index.html#SECLINKS
http://www.sans.org/webarchives.htm
http://www.sabernet.net/papers
http://www.bastille-linux.org
http://www.progressive-comp.com/Lists/?l=linux-security\
http://www.openbsd.org/papers/strlcpy-paper.ps
http://www.openbsd.org/papers/strlcpy-slides.ps
http://www.linuxsecurity.com
MAIL ADDRESSES:
<netbug@ftp.uk.linux.org> (for reporting of NetKit bugs)
MAIL LISTS:
<linux-security@redhat.com>
<suse-security@suse.com>
<bugtraq@netspace.org>
<xforce@iss.net>
<pam-list@redhat.com> (for Pluggable Authentication Module
info)
RFC:
rfc 1244
rfc 2196 obsoletes RFC 1244
MAILING LIST ARCHIVE:
http://lists.nas.nasa.gov/archives/ext/linux-security-audit/
http://www2.merton.ox.ac.uk/~security/ (also has some Bugtraq
and linux-security as well)
MISC SECURITY RELATED ARCHIVES:
http://www.phrack.com/Archives/
-------------------------------------------------------------------------
Subject: 1.6) What sort of thing is considered "off topic" for the list?
How do I configure my machine to be secure?
How do I set up ipfwadm to reject packets from a specific
host?
BIOS and boot security prior to Linux being loaded.
Where do I find the current release of "bind" for XXX Linux?
-------------------------------------------------------------------------
Subject: 1.7) What programs have been "fixed" by the audit?
A partial list of programs that have had bugs identified in them is:
dosemu
ncurses
slang (setfsuid issue - current slang already fixed $TERM)
termcap
metamail
tin
bsd games
elm
Note: this list is old and there have been MANY MANY more bugs found and
fixed by the LSAP. The LSAP is your friend, the LSAP knows all, the
LSAP wants to know if you have found any communists. (Paranoia reference
for those of you who don't get it :) ) There is a more complete list
available on: http://ferret.lmh.ox.ac.uk/~chris/fixed.txt
-------------------------------------------------------------------------
2. Bugs and what to do about them.
-------------------------------------------------------------------------
Subject: 2.1) I have found a bug, what should I do?
The first thing you should do is try to verify the bug on other
systems, making sure that the bug is actually what you think it is. The
next thing is you should report the bug to the programmer(s) or
maintainer of the software. Reporting the bug to the maintainer does
not gather as much "glory" for being an 3r334 (elite) cracker, but is the
responsible thing to do. If this does not work see Subject: 2.2 below.
When you report the bug to the programmer(s) you are giving them a
chance to repair the bug before it becomes general knowledge. When you
just release the bug to the public, you are actually endangering
systems. The amount of endangerment is dependent on the severity of the
bug.
When reporting a bug please be complete on your report, mention OS
version, system information, what you found, how you verified it and
other information in a similar vein.
If you can, sending a patch will be appreciated.
-------------------------------------------------------------------------
Subject: 2.2) I reported a bug to the programmer(s) and never got an
answer, what should I do now?
Now it is time to start considering either patching the program
yourself, or other measures. If you do not feel comfortable patching
yourself (or cannot due to licensing restrictions), about the only thing
you can do is release it to the public. The best venue to start with is
a smaller venue (such as this list), as some people on this list may be
able to exert a little more pressure on the person (or may actually know
or BE the person).
If this fails, you can release to bugtraq. This one is almost SURE to
get a response. It is a pretty sure way to get a response but is also a
very irresponsible first action. Please try to exhaust all other
resources and methods before doing this.
-------------------------------------------------------------------------
Subject: 2.3) I think this is a bug, but I am not sure. What should I
do?
This one is a little more tricky than the reported bug problem. You
have to try to strike a balance between exposing an actual bug and
giving enough information to get some help with figuring it out. It
helps if you are succinct. You can also just ask for people who know
about what you think is a bug and maybe take it "private."
The idea is not to give the bad guys a shot at exploiting the bug
before a fix is available.
-------------------------------------------------------------------------
Subject: 2.4) What kind of things should I be looking for?
First an administrative note: If you know of anythings that should be
being looked for in general and they are not on this list please email
me <demit@best.com>
You should take a look at:
http://www.pobox.com/~kragen/security-holes.html for an idea of what
sort of thing you should be looking for. Note: this is not a complete
document (as mentioned in the document itself.) Kragen is probably open
to suggestions on improvements to this so even if you don't need it, you
may wish to take a look at it and help improve it.
ADMIN NOTE: Thanks to Dr. Brand for input on many things below.
A partial list of what should be being looked for is:
/tmp race conditions (unsafe /tmp/blah-foo$$ opens)
If a program creates a file in /tmp (or other world/group-writable
directories) and then uses it later, it might be stolen or exchanged for
another one by the attacker with right timing. Examples have been files
created SUID +rwx for others that are stolen and filled in with a copy
of /bin/sh, files created and then given privileges that were exchanged
for copies of a shell that ended up privileged, files that a process
fills with data that are then exchanged for doctored files before the
data is used. Can be prevented by sticky bit on the directory itself,
but beware if the whole directory can be moved and replaced (i.e., the
parent can be written by anyone, or replaced by a symlink).
The following is from Kragen:
The largest class of /tmp races I've seen lately come from creating a
new file in /tmp with open() without using O_EXCL, which means that if
there's a file already there -- perhaps owned by someone else or with
lousy permissions -- it will be used, and can therefore be used to (a)
read private information (b) write untrustworthy information. Sticky
bit doesn't help a bit here.
This class of bugs can be reduced somewhat by using a special open()
that checks to see if we're writing into a world-writable directory,
and if so, sticks O_EXCL on the open().
Passing sensitive information between processes
Symlink following
This is the case where somebody runs a SUID/SGID program making one of
its outputs a symlink to a file to be overwritten/damaged. e.g.,
somehow get /var/log/messages to be a symlink to /etc/passwd will make
the system unusable (or in some cases force authentication to fall back
on a secondary mechanism that may be undesired (such as root in nis));
or a specially contrived syslog message could replace the passwd file
with a UID=0 account. If done to an input, it might enable the attacker
to read files that should be protected (e.g., /etc/shadow).
Buffer overruns (especially in suid/sgid processes)
This involves writing over the end of some buffer, thus damaging other
variables that happen to be defined before. In the worst cases, this
can lead to overwriting the return address of the function containing
the buffer (assuming the stack of the machine grows downwards, as it
almost always does), either causing a crash or getting the affected
function to "return" to somewhere else, like the start of the
'exec("/bin/sh", ...)' that is in libc for system(3) (giving a shell) or
something similar that the attacker could build up himself. This is
usually called "stack smashing." Typical attacks involve overlong
strings that are written into buffers without length checking. Many
functions in the C library don't check for buffer lengths, and are thus
exploitable. Examples include strcpy(3) strcat(3), sprintf(3),
getwd(3), gets(3). There are replacement functions available that do
check the lengths of what they are writing: strncpy(3), strncat(3),
snprintf(3) (or use the format to limit the length), getcwd(3),
fgets(3). At least glibc-2.0.94 does complain on linking if several of
the dangerous functions are used in a program. The right solution is
probably not just to rely on the cutoff done by functions that limit the
writing into buffers, but to check beforehand since a string mutilated
by chopping off its end might do other damage elsewhere.
Bad permissions on conf and other files (world/group writable)
Things like a world writable /etc/passwd are VERY dangerous. If you do
not know why I would suggest getting UNIX and Internet Security 2nd
edition from O'Reilly and Associates.
Not releasing suid/sgid when no longer needed
Not zeroing memory (such as passing plain text passwords)
Buffer overflows from untrusted sources
Examples: Environment variables, input from a network, program
arguments, etc) which may be passed to the following functions:
(including but not limited to):
sprintf, strcat, strcpy, scanf, memcpy, [^f]gets, open
creat, system, popen, exec.p, s[gu]id, getenv, chown
chmod
You can try throwing random streams of characters or X
events at applications and see if they crash or hang.
There is a study at:
<http://www.cs.wisc.edu/~bart/fuzz/fuzz.html>;
You can do chattr +a tmp and wait for a ton of programs
to scream at you that it cannot "unlink: operation not
permitted". (submitted by Chris Evans
<chris@ferret.lmh.ox.ac.uk>).
-------------------------------------------------------------------------
Subject: 2.5) Are there any tools that can help me check for bugs?
LCLint - Forces you to write code with annotations, which it then uses
to guide the checking of the code. If the code wasn't written with the
peculiar style and hand in hand with the annotations it demands,
you'll just get huge amounts of useless complaints. Some users report
that they were unable to write code for specific instances that did not
give errors. (thanks to Dr. Horst H. von Brand <vonbrand@inf.utfsm.cl>
for the information in this section (and other changes)). Available
from: ftp://larch.lcs.mit.edu/pub/Larch/lclint, last version (2.3i) Sep
1997. (This is a late alpha/early beta in Dr. Brands opinion.)
LCLint is also available at http://www.sds.lcs.mit.edu/lclint/
Slint (http://www.l0pht.com/slint.html)
A core product to be sold into an existing GUI development package.
Helps people be proactive while writing secure code by
highlighting potentional hot spots of exploitable routines and
poor memory allocations.
Identifies suspect blocks of code.
Makes the task of security review more paletable so you don't
need a team of high-level experts to go through megabytes of
code.
Supplies solutions and/or alternatives to problem areas.
Most security problems could have been fixed at the beginning
of development. Secure applications must start with a secure
base. The Best *BANG* for the buck is to be proactive at the
start of program creation.
Easy to implement into existing Y2K code review packages.
Slint is a commercial product of L0pht Industries.
gcc -Wall
Run the GNU gcc compiler with -Wall argument to enable all compiler
warning messages. This can help trap many problems in your code.
[e]grep ;-)
Auditd (http://www.hert.org/projects/linux/auditd/index.html)
Auditd is part of the linux kernel auditing toolkit. It will
capture auditing trails created by the kernel's auditing
facility from /proc/audit, filter them, and save them in
specific log files. For the moment, auditd only supports
the -t option which enables audit trails timestamping. Other
command line options will probably be implemented in the next
releases to add more flexiblity to the package. Also
available via ftp at ftp.hert.org /pub/linux/auditd. PGP
sigs and keys available as well.
-------------------------------------------------------------------------
Subject: 2.6) Where are some secure programming references?
You can find some secure programming references at:
Battening down the hatches: How can you implement mainframe-style access
control in Unix?
http://www.sun.com/sunworldonline/swol-04-1998/swol-04-unixsecurity.html
Designing secure software: A methodology for avoiding the security holes
that drive you mad.
http://www.sun.com/sunworldonline/swol-04-1998/swol-04-security.html
Security Code Review Guidlines.
http://www.homeport.org/~adam/review.html
Writing Safe Setuid Programs
http://olympus.cs.ucdavis.edu/~bishop/secprog.html
Shifting the Odds: Writing (More) Secure Software
http://www.research.att.com/~smb/talks/odds.[ps|pdf]
How to Find Security Holes
http://www.dnaco.net/~kragen/security-holes.[txt|html]
Secure UNIX Programming FAQ
http://www.whitefang.com/sup
Chapter 22 in "Practical UNIX & Internet Security" is called "Writing
Secure SUID and Network Programs".
"Writing Solid Code", published by Microsoft Press (I forget the
author). The book actually focuses on writing bug-free software, and
not on security issues, but there's definitely a large overlap there.
Take the SANS course on security programming taught by Matt Bishop.
It is very highly rated by those that have attended.
See www.sans.org for locations and dates.
Thanks to:
Tom Hall <thall@redrose.net>
Marko Milivojevic <M.Milivojevic@f.bg.ac.yu>
Wilson Roberto Afonso <wilson@zaz.com.br>
Joseph Pung <Pungj@meijer.com>
Doug Hughes <Doug.Hughes@Eng.Auburn.EDU>
Kragen <kragen@pobox.com>
Steven M. Bellovin <smb@research.att.com>
Aleph One <aleph1@dfw.net>
Chris Evans <chris@ferret.lmh.ox.ac.uk>
Dr. Horst H. Von Brand <vonbrand@inf.utfsm.cl>
-------------------------------------------------------------------------
3. Auditing tasks.
-------------------------------------------------------------------------
Subject: 3.1) How do I suggest something I feel needs auditing but do
not have time to do myself?
You can suggest it to the list for possible review by auditing members.
Things will be worked on by people as they have the time. Please bear
in mind that this security project is HUGE and is not really limiting
itself to the linux community, we are using linux as the os we are
fixing things on but the fixes are going into the programs and not just
the kernel. If you have seen no action for maybe a month (or if you
feel this is a critical bug (for instance one that allows root access,)
you can contact me at jeff@kodenkan.com or demit@best.com directly, I'll
attempt to route it to whoever is handling things of this sort,) you
can repost it to the list.
-------------------------------------------------------------------------
Subject: 3.2) How can I find something that needs auditing and is not
currently being worked on by anyone?
You will have to follow the list for suggestions of what to do. If you
are really eager to get started, just post a message to the list asking
what people would like to see done. Alternatively, you can follow the
list and wait until someone asks for a package to be audited or for
someone else to ask for help in their audit.
-------------------------------------------------------------------------
4. Other security information.
-------------------------------------------------------------------------
Subject: 4.1) What about encrypted file systems?
There are various in progress encrypted file systems some of which you
can find detailed here:
CFS: ftp://ftp.research.att.com/dist/mab (read the README file)
Reportedly this encrypts directories as well.
Transparent Cryptographic Filesystem:
http://tcfs.dia.unisa.it/
has PAM support for auth as well as support for encrypted
NFS.
DES encryption with loopback:
ftp://ftp.csua.berkeley.edu/pub/cypherpunks/filesystems/linux
Patches for DES encryption with loopback:
ftp://ftp.is.co.za/linux/local/kernel/crypto/\
loopback-device-berkeley-recent/
Patches for 2.1 series kernels:
ftp://ftp.is.co.za/linux/local/kernel/crypto/\
loopback-device-aem/
SFS (Stegano File System):
http://www.linux-security.org/sfs
-------------------------------------------------------------------------
Subject: 4.2) What are some automatic tools I can use to improve
security?
There are a number of things that will "automatically" help to improve
security both while programming and on systems.
These include (but are not limited to):
StackGuard
This tool is specifically designed to help defend against stack
smashing attacks. You can also find a copy of RedHat that is stack
guarded here. Available at:
http://immunix.org
Secure-Linux Patch
This patch from Solar Designer for the 2.0.x kernel series makes exploits
for buffer overflows much harder, and prevents /tmp link attacks completely.
This should be patched into all kernels of security sensitive linux servers.
http://www.false.com/security/linux
There is also one for the 2.2.x (x>=12) kernel at
http://www.openwall.com/linux) as well.
Bounds-checking gcc
This compiler does full bounds-checking of arrays in C. Not very
stable or mature but relevant. Available at:
http://www-ala.doc.ic.ac.uk/~phjk/BoundsChecking.html
Electric fence, checker, ccmalloc, and mpr are available on
sunsite.unc.edu in /pub/Linux/devel/lang/c:
Electric Fence:
Electric Fence is a different kind of malloc() debugger. It uses the
virtual memory hardware of your system to detect when software overruns
the boundaries of a malloc() buffer. It will also detect any accesses
of memory that has been released by free(). Because it uses the VM
hardware for detection, Electric Fence stops your program on the first
instruction that causes a bounds violation. It's then trivial to use a
debugger to display the offending statement.
checker:
checker - Memory access debugger for C language development
Drop-in addition to `gcc' that allows programmers to find most
memory-related bugs easily. Checker automatically finds:
* null pointer dereferences (read, write, and execute accesses)
* writes to read-only memory
* accesses to free blocks (read, write, and execute)
* reads/writes to uninitialized bytes (in automatic and dynamic
memory)
* reads/writes to "red zones" (in automatic and dynamic memory)
* reads/writes/executes outside memory segments
* free called with address not obtained from malloc
* free an already freed block
* ...and many more!
Checker's main disadvantage is that it causes the program to run many
times slower. You can compile your program to use Checker by using
the command `checkergcc' in place of `gcc'. Please do read the
documentation, it contains some important information.
ccmalloc:
ccmalloc - A memory profiler/debugger
This is a memory profiling package. It can be used to debug various
memory allocation problems, including:
o memory leaks
o multiple deallocation of the same data
o under writes and over writes
o writes to already deallocated data
mpr:
mpr can be used to find malloc/realloc memory leaks and memory
allocation statistics and patterns - it does not find memory
corruption errors. It uses a simple, brute force strategy - log all
memory alloc/free requests to a file and then post-process this log
file once the program has terminated.
mpr keeps track of the entire call chain leading up to a memory
alloc/free request, making it much easier to find memory allocation
problems. This is superior to conventional methods of keeping track of
only the immediate caller of a memory alloc/free routine, e.g. using
__FILE__ and __LINE__.
dmalloc:
http://www.letters.com/dmalloc/
The debug memory allocation or dmalloc library has been designed as a
drop in replacement for the system's malloc, realloc, calloc, free and
other memory management routines while providing powerful debugging
facilities configurable at runtime. These facilities include such
things as memory-leak tracking, fence-post write detection, file/line
number reporting, and general logging of statistics.
The library is reasonably portable having been run successfully on at
least the following operating systems: AIX, BSDI, DG/UX, FreeBSD, HPUX,
Irix, Linux, MS-DOG, NeXT, OSF, SCO, Solaris, SunOS, Ultrix, Unixware,
Windows and even Unicos on a Cray Y-MP. It also provides support for
the debugging of threaded programs.
The package includes the library, configuration scripts, debug utility
application, test program, and extensive documentation (formats: text,
texi, info, and ps).
Multi-stack allocator (technique for ecgs) (patch and run-time lib):
"Multi-stack" is a technique of allocating local arrays to prevent
exploits for most of the buffer overflows. It is NOT a complete
solution for buffer overflows.
http://www.ipmce.su/~sorlov/security.html
--
Jeffery Graham - Senior Systems Administrator (UNIX)
PGP public key available by request