Enabling realtime logging in Toil
Cactus Command: /private/home/hickey/dev/cactus.maf/venv-cactus.maf/bin/cactus-maf2bigmaf ./js-bb ./16-t2t-apes-2023v2.hs1.maf.gz ./16-t2t-apes-2023v2.hs1.bigmaf.bb --refGenome hs1 --halFile ./16-t2t-apes-2023v2.hal --logFile ./16-t2t-apes-2023v2.hs1.bigmaf.bb.log --batchLogsDir batch-logs-8apes --coordinationDir /data/tmp --batchSystem slurm
Cactus Commit: a8bd77e65d7f7c26fd7a6d69a110d1fe23b275c9
Importing ./16-t2t-apes-2023v2.hs1.maf.gz
Importing ./16-t2t-apes-2023v2.hal
Batch system does not support auto-deployment. The user script ModuleDescriptor(dirPath='/private/groups/cgl/hickey/work/cactus.maf/src', name='cactus.maf.cactus_maf2bigmaf', fromVirtualEnv=False) will have to be present at the same location on every worker.
Running Toil version 6.0.0-0e2a07a20818e593bfdfde3cc51ca4ad809fde96 on host phoenix.prism.
Starting real-time logging.
Issued job 'maf2bigmaf_workflow' kind-maf2bigmaf_workflow/instance-l1or95xk v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
1 jobs are running, 0 jobs are issued and waiting to run
2024-02-20 12:26:59.853921: Running the command: "mafToBigMaf"
2024-02-20 12:26:59.871909: Running the command: "bedToBigBed"
2024-02-20 12:26:59.877594: Running the command: "hgLoadMafSummary"
Issued job 'maf2bigmaf_chrom_sizes' kind-maf2bigmaf_chrom_sizes/instance-aa5n5u6j v1 with job batch system ID: 2 and disk: 32.1 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
Reading HAL file from job store to /data/tmp/38286b9919805daeb32423b1ab33a70b/8acc/6adf/tmp0f5ms2_7/16-t2t-apes-2023v2.hal
Computing chromosome sizes
2024-02-20 12:28:56.675593: Running the command: "halStats /data/tmp/38286b9919805daeb32423b1ab33a70b/8acc/6adf/tmp0f5ms2_7/16-t2t-apes-2023v2.hal --chromSizes hs1"
2024-02-20 12:28:56.686162: Successfully ran: "halStats /data/tmp/38286b9919805daeb32423b1ab33a70b/8acc/6adf/tmp0f5ms2_7/16-t2t-apes-2023v2.hal --chromSizes hs1" in 0.0098 seconds and 8.4 Mi memory
2024-02-20 12:28:56.686282: Running the command: "halStats --genomes /data/tmp/38286b9919805daeb32423b1ab33a70b/8acc/6adf/tmp0f5ms2_7/16-t2t-apes-2023v2.hal"
2024-02-20 12:28:56.739423: Successfully ran: "halStats --genomes /data/tmp/38286b9919805daeb32423b1ab33a70b/8acc/6adf/tmp0f5ms2_7/16-t2t-apes-2023v2.hal" in 0.0526 seconds and 16.3 Mi memory
Got message from job at time 02-20-2024 12:28:57: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-maf2bigmaf_chrom_sizes/instance-aa5n5u6j/cleanup/file-395666c14a7b4cbe87a18c04786010db/stream used 100.00% disk (32.1 GiB [34454147072B] used, 32.1 GiB [34454088035B] requested).
Issued job 'maf2bigmaf' kind-maf2bigmaf/instance-wgfo7qj6 v1 with job batch system ID: 3 and disk: 36.2 Gi, memory: 4.0 Gi, cores: 1, accelerators: [], preemptible: False
Issued job 'maf2bigmaf_summary' kind-maf2bigmaf_summary/instance-q_pg11hu v1 with job batch system ID: 4 and disk: 10.3 Gi, memory: 4.0 Gi, cores: 1, accelerators: [], preemptible: False
Reading MAF file from job store to /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/16-t2t-apes-2023v2.hs1.maf.gz
Reading MAF file from job store to /data/tmp/38286b9919805daeb32423b1ab33a70b/d055/4e8a/tmp3ld2h3yi/16-t2t-apes-2023v2.hs1.maf.gz
2024-02-20 12:29:21.464654: Running the command: "bash -c set -eo pipefail && gzip -dc /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/16-t2t-apes-2023v2.hs1.maf.gz | sed -f /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/genome_to_alnum.sed | mafDuplicateFilter -km - | hgLoadMafSummary -minSeqSize=1 -test hs1 bigMafSummary stdin"
2024-02-20 12:29:21.464748: Running the command: "bash -c set -eo pipefail && gzip -dc /data/tmp/38286b9919805daeb32423b1ab33a70b/d055/4e8a/tmp3ld2h3yi/16-t2t-apes-2023v2.hs1.maf.gz | mafDuplicateFilter -km - | mafToBigMaf hs1 stdin stdout | sort -k1,1 -k2,2n"
2024-02-20 12:51:16.847603: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /data/tmp/38286b9919805daeb32423b1ab33a70b/d055/4e8a/tmp3ld2h3yi/16-t2t-apes-2023v2.hs1.maf.gz | mafDuplicateFilter -km - | mafToBigMaf hs1 stdin stdout | sort -k1,1 -k2,2n'" in 1315.3816 seconds and 10.9 Mi memory
2024-02-20 12:51:16.847839: Running the command: "bedToBigBed -type=bed3+1 -as=/data/tmp/38286b9919805daeb32423b1ab33a70b/d055/4e8a/tmp3ld2h3yi/bigMaf.as -tab /data/tmp/38286b9919805daeb32423b1ab33a70b/d055/4e8a/tmp3ld2h3yi/bigMaf.bed /data/tmp/38286b9919805daeb32423b1ab33a70b/d055/4e8a/tmp3ld2h3yi/hs1.chrom_sizes /data/tmp/38286b9919805daeb32423b1ab33a70b/d055/4e8a/tmp3ld2h3yi/16-t2t-apes-2023v2.hs1.bigmaf.bb"
2024-02-20 12:51:51.592464: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/16-t2t-apes-2023v2.hs1.maf.gz | sed -f /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/genome_to_alnum.sed | mafDuplicateFilter -km - | hgLoadMafSummary -minSeqSize=1 -test hs1 bigMafSummary stdin'" in 1350.1263 seconds and 2.8 Gi memory
2024-02-20 12:51:51.592904: Running the command: "bash -c set -eo pipefail && cut -f2- bigMafSummary.tab | sed -f /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/alnum_to_genome.sed | sort -k1,1 -k2,2n"
2024-02-20 12:52:00.411413: Successfully ran: "bash -c 'set -eo pipefail && cut -f2- bigMafSummary.tab | sed -f /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/alnum_to_genome.sed | sort -k1,1 -k2,2n'" in 8.8178 seconds and 8.0 Mi memory
2024-02-20 12:52:00.411621: Running the command: "bedToBigBed -type=bed3+4 -as=/data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/mafSummary.as -tab /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/summary.bed /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/hs1.chrom_sizes /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/16-t2t-apes-2023v2.hs1.bigmaf.summary.bb"
2024-02-20 12:52:08.519599: Successfully ran: "bedToBigBed -type=bed3+4 -as=/data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/mafSummary.as -tab /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/summary.bed /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/hs1.chrom_sizes /data/tmp/38286b9919805daeb32423b1ab33a70b/6305/fa5e/tmpo9cwi8at/16-t2t-apes-2023v2.hs1.bigmaf.summary.bb" in 8.1073 seconds and 23.8 Mi memory
2024-02-20 13:16:37.282396: Successfully ran: "bedToBigBed -type=bed3+1 -as=/data/tmp/38286b9919805daeb32423b1ab33a70b/d055/4e8a/tmp3ld2h3yi/bigMaf.as -tab /data/tmp/38286b9919805daeb32423b1ab33a70b/d055/4e8a/tmp3ld2h3yi/bigMaf.bed /data/tmp/38286b9919805daeb32423b1ab33a70b/d055/4e8a/tmp3ld2h3yi/hs1.chrom_sizes /data/tmp/38286b9919805daeb32423b1ab33a70b/d055/4e8a/tmp3ld2h3yi/16-t2t-apes-2023v2.hs1.bigmaf.bb" in 1520.4331 seconds and 233.4 Mi memory
Got message from job at time 02-20-2024 13:16:47: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-maf2bigmaf/instance-wgfo7qj6/cleanup/file-30bb786edb1b40cb99c04bffbb1f9bf2/stream used 152.81% disk (55.3 GiB [59390197760B] used, 36.2 GiB [38864817313B] requested).
Finished toil run successfully.
Stopping real-time logging server.
Joining real-time logging server thread.
Exporting file:///private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/16-t2t-apes-2023v2.hs1.bigmaf.bb
Exporting file:///private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/16-t2t-apes-2023v2.hs1.bigmaf.summary.bb
Successfully deleted the job store: FileJobStore(/private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/js-bb)
cactus-maf2bigmaf has finished after 3012.209180281963 seconds