IBU
Consol
|
|
Index
docs . freebsd . org / en / books / handbook / mirrors /
#external-mirrors
"A.4.1. Introduction
As of December 2020, FreeBSD uses git as the primary version
control system for storing all of FreeBSD's source code and
documentation. Changes from the git repo on the stable/11,
stable/12 and related releng branches are exported to the
subversion repository. This export will continue through the
life of these branches. From July 2012 to March 2021, FreeBSD
used Subversion as the only version control system for
storing all of FreeBSD's Ports Collection. As of April 2021,
FreeBSD uses git as the only version control system for
storing all of FreeBSD's Ports Collection."
Corrections Pending Integration @ 2019-09-14:
- Although CTM's etymology from first authour was "Cvs Tree
Mirror" that now misleads, per my
http://www.berklix.org/ctm/#what "Code Through Mail" might
best summarise it at 2018-12
- CTM is now a means of delivering tree updates (Similar
result to to rdist6 & rsync & mirror, though those
rely on different functionality: end to end live pull using
ports some firewalls etc may block, whereas CTM just pushes
with SMTP, unlikely blocked)
- Most trees we deliver with CTM are src-* releases, + 1
ports, + 1 SVN, but surely CTM could just as readily deliver
a GIT tree as an SVN, as its already done for CVS too ?
- Admittedly if freebsd moves to GIT, we'd need to tweak
the server scripts to do GIT export instead of svn
export.
- My http://www.berklix.org/ctm/#why "Why Use CTM for
Delivery Instead of SVN ?" may also misleading, accidentaly
reinforcing the wrong impression that CTM is just an
alterantive to SVN. Its not. I'll fix it.
-
2022-07-30 Status Thanks to Andre for reporting, & Warner Losh for
fixing the up-stream git to svn exporter that CTM depends
on.
/pub/FreeBSD/development/CTM/svn-cur
1124 Jul 2 00:05 svn-cur.10151.xz
232956 Jul 30 00:06 svn-cur.10152.xz
/pub/FreeBSD/development/CTM/src-12
755 Jul 2 00:05 src-12.1491.gz
172904 Jul 30 00:07 src-12.1492.gz
-
2021-07-30 Status
ftp://
ftp . berklix . org / pub / FreeBSD / development /
CTM / |
Date Of Last Delta |
Status
Scripts & mail list in place for
all
|
src-4/src-4.2355.gz |
Nov 15 2014 |
src-5/src-5.0862.gz |
Nov 15 2014 |
src-6/src-6.1661.gz |
Nov 12 2015 |
src-7/src-7.1772.gz |
Jul 1 2020 |
src-8/src-8.1980.gz |
Jul 1 2020 |
src-9/src-9.1930.gz |
Aug 11 2020 |
src-10/src-10.2188.gz |
Dec 17 2020 |
src-11/src-11.1937.gz |
Jul 28 08:11 |
Still being exported from git to svn |
src-12/src-12.1205.gz |
Jul 30 08:09 |
Still being exported from git to svn |
src-13/ |
|
The CTM scripts & mail list in place, but I
recall never exported from git to svn |
src-cur/src-cur.14812.gz |
Dec 20 2020 |
No longer being exported from git to svn ? |
ports-cur/ports-cur.13927.gz |
Mar 31 16:34 |
No longer being exported from git to svn ? |
svn-cur/svn-cur.09751.xz |
Jul 30 08:05 |
- 2021-03: docs
. freebsd . org / en / books / handbook / mirrors / #git
"After March 2021, subversion use is only for legacy
branches (stable/11 and stable/12)."
Note ctm generation is depedent on extracting suning SVN,
none of us have had time to convert extraction to git, if you
have time to convert our scripts from svn to git, please
discuss on mail list ctm-users@freebsd
.org
- Discussion Mail Lists Carry
Latest Info
-
2020-12-20 post to ctm-users@freebsd
.org
2021-01-05: ftp:// ftp . berklix
. org / pub / git / README
To view directory: ftp:// ftp . berklix . org
/ pub / git/
-
NEW ADDITIONAL SERVICE: At 2019-08-16 04:00+0200
-
2019 August Outage, Status At 2019-08-16 04:00+0200 : On
freebsd.org: all fixed now as far as we know except ctm-src-cur still delayed by
moderation, & many others not checked if moderated.
- OK: ctm-
src-11-fast, ctm-src-12, ctm-src-12-fast, ctm-src-cur-fast, ctm-svn-cur-fast, ctm-
ports-cur.
- Unchecked, likely OK: ctm-
src-4, ctm-
src-4-fast, ctm-
src-5, ctm-
src-5-fast, ctm-
src-6, ctm-
src-6-fast, ctm-
src-7, ctm-
src-7-fast, ctm-
src-8, ctm-
src-8-fast, ctm-
src-9, ctm-
src-9-fast, ctm-
src-10, ctm-
src-10-fast, ctm-
src-11, ctm-
src-11-fast, ctm-
src-cur, ctm-svn-cur.
- The delta generator host
died.
- Generation moved to ctm.berklix.org
(which already had ftp archives).
- Last backup of generator build scripts was
old,
- Generator
build scripts were re-updated,
- Generator trees built
&
- Deltas generation
started.
- Briefly deltas were received on lists
,
- When ctm-src-12 generation started,
it sent with From: owner-ctm-src-12[-fast]@mailman.berklix.org
, which was confusingly similar to mailman
standard alias ctm-src-12[-fast]-owner@ , so
for ctm-src-12 it was
changed to From:
freebsd-ctm-bld@berklix
- Then although ctm.berklix.org was
mailing deltas (other than ctm-src-12) to lists on
freebsd.org, after a bit, nothing further
appeared on lists
- Guessed that mail header From: address
did not match what @freebsd.org lists
allowed.
- Guessed something had happened the
freebsd.org end;
- The CTMownermail values in the Mail List Config
Files that generate the From: for the
other lists, had not changed, so those non ctm-src-12 lists should
have continued working;
- Learnt how to specify the From:
field,
- The question was: what address had
been previously used ? Various guesses failed
experimenting with old addresses including
elements cauchy, library, etc. .
- Gave up guessing, subscribed a new
From address freebsd-ctm-bld@berklix to all ctm
lists.
- Guessed it would be by default
moderated.
-
Asked postmaster@freebsd to un-moderate
that address on all ctm lists. Some or most
were unmoderated but at least one was not, as
this was seen for
ctm-src-cur:
- Created on ctm.berklix.org UTC:
Aug 12 14:08 src-cur.14144.gz
-
Outgoing tap to jhs-test@:
- Date: Mon, 12 Aug 2019
14:08:02 GMT
- From:
freebsd-ctm-bld@berklix
- To: ctm-ctm-src-cur@mailman.berklix
- Subject: ctm-mail src-cur.14144.gz
1/1
-
Moderated:
- Subject: Your message to
ctm-ctm-src-cur
awaits moderator approval
- From: ctm-ctm-src-cur-owner@freebsd
- Date: Mon, 12 Aug 2019
14:08:24 +0000
- To:
freebsd-ctm-bld@berklix
- Your mail to 'ctm-ctm-src-cur' with the
subject ctm-mail src-cur.14144.gz
1/1
- Is being held until the list
moderator can review it for
approval.
- Received from list on jhs@ host
UTC: Aug 12 14:10 src-cur.14144.gz
- ctm-svn-cur: the tap on
ctm.berklix.org mail outgoing showed it ignoring
the directive set CTMownermail
freebsd-ctm-bld@berklix in
ctm.berklix.org/~sms/library/ctm/ctm_generation/ctm_conf.svn-cur
copy here:
www . berklix . com / ~jhs / src / bsd / fixes /
freebsd / ports / gen / misc / ctm / files /
ftp.berklix.org / ctm_conf.svn-cur so
freebsd.org rejects those
Another script was fixed
.
-
Unchecked: A
test was run on all lists, to
ensure all have been unmoderated (as some of
the old lists rarely carry traffic, so cannot
be observed, but might someday receive a delta
we want passed).
Content was produced by this
#!/bin/csh
foreach i ( `cat tests_lists.txt` )
echo "To: $i" > tests.now
cat tests_body.txt >> tests.now
sendmail -i -t < tests.now
echo Done $i
sleep 3
end
None of those test have arrived back
from any lists, so we don't know if all lists
have been un-moderated.
- 2019-09-23 Removed tap to jhs-test@ from
ctm-src-cur & ctm-src-cur-fast
|
- jhs & sms: Optimise crontab entries, & publicly
document so users know when it runs.
- Maximum length in kilobytes (KB) of a message body
max_message_size = 300 by default. Either jhs@ can increas if
it goes over limit, or jhs@ can consult sms@ what limits
should be ? Maybe higher limits for fast streams ?
- jhs or sms: Backup of generator
build scripts
to be adjacent to Mail List Config
Files
-
Write a brief intro on how delta generation works, (in case
something happens to Stephen, Julian, or
ctm.berklix.org)
Odds About
SVN To Move:
-
Request a
CTM recipient to: add in to
manual
Why Use
CTM for Delivery Instead of
SVN ?
- (
CTM can also deliver you an
SVN tree, so you can still use
SVN locally.)
- If you have a slow net connection aka limited bandwidth,
or intermittent connectivity eg rarely reliably in one place
long enough to complete an
SVN update of whole trees. Or expensive data rates eg
mobile tariffs.
- If your corporate/ institutional firewall blocks you, but
allows mail.
- The patches are compressed, saving time & data
transfer charges.
- Imagine you'r travelling/ on holiday, you don't fancy
paying heavy data charges tethering via your mobile phone, so
stroll into a coffee shop with WLAN, or visit somewhere with
an incomprehensible maybe foreign language documented fire
walled WLAN, or if lucky encounter a town centre open WLAN
(there's a few in central Munich), but you have nowhere to
sit, no power socket, & you want it all downloaded as
fast as possible, not blocked on unknown ports, no thinking
on the hoof required, combined with mail pick up you know
won't be blocked:
CTM has it all rolled up ready for you. You don't even
have to stand with no seat, consuming battery, applying all
the deltas.
CTM deltas can be applied later where you do have a seat
& power supply.
- Imagine you'r in an airport departure lounge, with WLAN,
but may be called any time. So you just want those deltas
downloaded Fast. You can apply them later, perhaps even on
the plane/ train/ hotel with no net.
- A unique reproducible trees with eg /usr/svn/.ctm_status,
additional to the .svn status
- Pre extracted trees for all versions of src & ports
as well as
SVN tree available via
CTM delivery, so you can do local
SVN searches, exports etc.
- Suggestion: You may want to subscribe a different mail
address for
CTM delta reception, so you can first specify eg
fetchmail fetches your human mail.)
- Errors or Omissions ? Mail page author and/
or list ctm-users@freebsd.org
- "Code Through Mail" might best summarise it at
2018-12
-
man 1 ctm,
man 5 ctm,
man 1 ctm_smail,
man 1 ctm_rmail,
man 1 ctm_dequeue
- Source Code (patches, delivery & update)
Through Mail.
CTM is ideal for receiving source code updates on slow,
intermittent, or expensive tarif (eg mobile/cell) internet
connections. Used by & supported on FreeBSD (FreeBSD also supports
& is supported by other access methods eg
rsync);
CTM could easily be used by other BSDs, Linuxes &
beyond etc.
- Etymology: "Cvs Tree Mirror" as named by code author phk@ in
1994-04-11
(
CVS is a source code repository management system, as are
eg SCCS,
RCS,
SVN/Subversion.
BSD used SCCS,
FreeBSD used CVS, although FreeBSD now (2018) uses
SVN,
CTM continues to deliver FreeBSD Code Through Mail.
- Started by providing Deltas with public ftp access, (When
freebsd.org decided to cease to host the master deltas sent
from the generator site.
(& thus we also lost all out mirror sites).
- A set of
CTM Generator Build Scripts
& trees, ready to generate deltas
were copied in case anything failed at Missouri build
site.
- ctm-src-12 mail list created on
ctm.berlix.org
- 2019, The missouri master host died & production
moved to ctm.berlix.org
- Please do not overload ftp://ftp.berklix.org. There are
no mirrors. If you want to offer a public mirror, please do,
please contact contact host
master or ctm-users@.
- Please avoid heavy traffic during work days &
business hours in Germany (winter time zone +01:00, summer
+02:00). Time
now At 2019-08 it's a lightly loaded host, but that may
change, & I dont want to annoy the host.
- There is no point regularly ftp'ing latest deltas from here, as a purpose of
CTM is to provide Push technology, so you should be
receiving distribution updates by ctm-* mail lists.
If you want to regular fetch (which is Pull technology, not
push) then you should not be using
CTM, but be using
SVN or
Rsync direct form a freebsd.org mirror, nor from
berklix.org
Just use berklix.org ftp archives if you fail to receive a
delta via
ctm_rmail.
-
Mail Lists
- ctm-announce@freebsd
Not Used Much Stephen (generator of deltas) prefers
to use ctm-users@ so users can reply.
Archive of announcements lists
. freebsd . org / pipermail / ctm-announce /
Join here:
lists . freebsd . org / mailman / listinfo /
ctm-announce
- ctm-users@freebsd
.org
Archive of problems, solutions, announcements lists .
freebsd . org / pipermail / ctm-users/
Join here: lists
. freebsd . org / mailman / listinfo / ctm-users
-
- Contains dead link:
http://www.freebsd.org/handbook/synching.html#CTM
-
Some topics:
-
lists .
freebsd . org / mailman / listinfo has lots of CTM delta distribution lists
- www
. berklix . com / ~jhs / dots / .procmailrc_lists
Examples of how to automatically process receipt from
these lists, applying the deltas to update a tree.
-
Documentation
- Dead link: http:// www . freebsd . org / handbook
/ synching . html # CTM
(in mail archive intro, not Href'd) URL redirects
to
Dead link: https:// www . freebsd . org / doc /
en_US . ISO8859 - 1 / books / handbook / synching .
html # CTM
- Dead URL: http:// www . freebsd . org / doc /
en_US . ISO8859 - 1 / books / handbook / ctm.html @
2016-04-23 & 2016-09-01
-
svnweb . freebsd . org / base / head / usr.sbin / ctm
/ ctm / ctm.1
/usr/src usr.sbin/ctm/ctm/ctm.1 point to this page,
kindly committed by brueffer@
Mon 14 Dec 2016 svn rev 292212 from a patch jhs@ sent.
-
Deltas For FreeBSD
- Stephen Montgomery-Smith runs the incremental delta
generator.
- ftp://
ftp . berklix . org / pub / FreeBSD / development / CTM
/
Deltas: 16.3 G (@ 2019-01-25. Includes base deltas
(names such as src-cur.13800xEmpty.gz ,
ports-cur.12800xEmpty.gz , ctm-svn-cur.07300xEmpty.xz ) as well as
incremental deltas (names such as src-cur.13882.gz ,
ports-cur.12833.gz , ctm-svn-cur.07398.xz ) of base deltas,
Updated regularly (@ 2019-01) with rsync from Master.
(Pre 2015, not sure if still the same) times were (TZ:
GMT(UTC)-06:00), starting at 23:00 GMT(UTC) =
(midnight German
(GMT(UTC)+01:00) time), ( maybe +/- an hour
when computers may implement Missouri time on master
& German summer times on slave mirror). (May
also vary later subject to traffic experience), (Not
directly hung after the master
CTM server runs (several times per day).
-
Patches For
CTM source itself
(Not to apply to generic large FreeBSD source trees of
src/ports/svn/) but patches to to apply to the small
CTM source itself. Patches to support newer releases
of FreeBSD
CTM streams.
(
CTM source lives within FreeBSD src/ & may (after
2018-10) migrate to ports/)
-
Generator Build Scripts &
Examples
-
berklix . com / ~jhs / src / bsd / fixes / freebsd / ports /
gen / misc / ctm / README.JHS
Full List of Delta Mail Lists & FTP
Archives on freebsd.org & ctm.berklix.org:
|