Enabling realtime logging in Toil
Cactus Command: /private/home/hickey/dev/cactus.maf/venv-cactus.maf/bin/cactus-maf2bigmaf ./js-bb ./16-t2t-apes-2023v2.GCA_028858775.2.maf.gz ./16-t2t-apes-2023v2.GCA_028858775.2.bigmaf.bb --refGenome GCA_028858775.2 --halFile ./16-t2t-apes-2023v2.hal --logFile ./16-t2t-apes-2023v2.GCA_028858775.2.bigmaf.bb.log --batchLogsDir batch-logs-8apes --coordinationDir /data/tmp --batchSystem slurm
Cactus Commit: a8bd77e65d7f7c26fd7a6d69a110d1fe23b275c9
Importing ./16-t2t-apes-2023v2.GCA_028858775.2.maf.gz
Importing ./16-t2t-apes-2023v2.hal
Batch system does not support auto-deployment. The user script ModuleDescriptor(dirPath='/private/groups/cgl/hickey/work/cactus.maf/src', name='cactus.maf.cactus_maf2bigmaf', fromVirtualEnv=False) will have to be present at the same location on every worker.
Running Toil version 6.0.0-0e2a07a20818e593bfdfde3cc51ca4ad809fde96 on host phoenix.prism.
Starting real-time logging.
Issued job 'maf2bigmaf_workflow' kind-maf2bigmaf_workflow/instance-88ol8qqp v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
1 jobs are running, 0 jobs are issued and waiting to run
2024-02-20 14:09:10.914894: Running the command: "mafToBigMaf"
2024-02-20 14:09:10.926927: Running the command: "bedToBigBed"
2024-02-20 14:09:10.932850: Running the command: "hgLoadMafSummary"
Issued job 'maf2bigmaf_chrom_sizes' kind-maf2bigmaf_chrom_sizes/instance-jdfp0l6z v1 with job batch system ID: 2 and disk: 32.1 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
Reading HAL file from job store to /data/tmp/81fefe459ed352c88c0799391327a7f5/35e9/9d32/tmpv7jaxg3x/16-t2t-apes-2023v2.hal
Computing chromosome sizes
2024-02-20 14:11:02.130738: Running the command: "halStats /data/tmp/81fefe459ed352c88c0799391327a7f5/35e9/9d32/tmpv7jaxg3x/16-t2t-apes-2023v2.hal --chromSizes GCA_028858775.2"
2024-02-20 14:11:02.167062: Successfully ran: "halStats /data/tmp/81fefe459ed352c88c0799391327a7f5/35e9/9d32/tmpv7jaxg3x/16-t2t-apes-2023v2.hal --chromSizes GCA_028858775.2" in 0.0355 seconds and 8.3 Mi memory
2024-02-20 14:11:02.167191: Running the command: "halStats --genomes /data/tmp/81fefe459ed352c88c0799391327a7f5/35e9/9d32/tmpv7jaxg3x/16-t2t-apes-2023v2.hal"
2024-02-20 14:11:02.219769: Successfully ran: "halStats --genomes /data/tmp/81fefe459ed352c88c0799391327a7f5/35e9/9d32/tmpv7jaxg3x/16-t2t-apes-2023v2.hal" in 0.0521 seconds and 16.2 Mi memory
Got message from job at time 02-20-2024 14:11:02: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-maf2bigmaf_chrom_sizes/instance-jdfp0l6z/cleanup/file-204cff7bbc52433d9a255907052f0bfc/stream used 100.00% disk (32.1 GiB [34454138880B] used, 32.1 GiB [34454088035B] requested).
Issued job 'maf2bigmaf_summary' kind-maf2bigmaf_summary/instance-wvt647ua v1 with job batch system ID: 3 and disk: 10.5 Gi, memory: 4.0 Gi, cores: 1, accelerators: [], preemptible: False
Issued job 'maf2bigmaf' kind-maf2bigmaf/instance-45z1s5pd v1 with job batch system ID: 4 and disk: 36.7 Gi, memory: 4.0 Gi, cores: 1, accelerators: [], preemptible: False
Reading MAF file from job store to /data/tmp/81fefe459ed352c88c0799391327a7f5/0432/2938/tmp74q_1x4k/16-t2t-apes-2023v2.GCA_028858775.2.maf.gz
Reading MAF file from job store to /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/16-t2t-apes-2023v2.GCA_028858775.2.maf.gz
2024-02-20 14:11:26.580052: Running the command: "bash -c set -eo pipefail && gzip -dc /data/tmp/81fefe459ed352c88c0799391327a7f5/0432/2938/tmp74q_1x4k/16-t2t-apes-2023v2.GCA_028858775.2.maf.gz | mafDuplicateFilter -km - | mafToBigMaf GCA_028858775.2 stdin stdout | sort -k1,1 -k2,2n"
2024-02-20 14:11:26.580175: Running the command: "bash -c set -eo pipefail && gzip -dc /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/16-t2t-apes-2023v2.GCA_028858775.2.maf.gz | sed -f /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/genome_to_alnum.sed | mafDuplicateFilter -km - | hgLoadMafSummary -minSeqSize=1 -test g7 bigMafSummary stdin"
2024-02-20 14:34:19.968564: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/16-t2t-apes-2023v2.GCA_028858775.2.maf.gz | sed -f /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/genome_to_alnum.sed | mafDuplicateFilter -km - | hgLoadMafSummary -minSeqSize=1 -test g7 bigMafSummary stdin'" in 1373.3873 seconds and 2.8 Gi memory
2024-02-20 14:34:19.968961: Running the command: "bash -c set -eo pipefail && cut -f2- bigMafSummary.tab | sed -f /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/alnum_to_genome.sed | sort -k1,1 -k2,2n"
2024-02-20 14:34:30.360977: Successfully ran: "bash -c 'set -eo pipefail && cut -f2- bigMafSummary.tab | sed -f /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/alnum_to_genome.sed | sort -k1,1 -k2,2n'" in 10.3913 seconds and 7.9 Mi memory
2024-02-20 14:34:30.361185: Running the command: "bedToBigBed -type=bed3+4 -as=/data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/mafSummary.as -tab /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/summary.bed /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/GCA_028858775.2.chrom_sizes /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/16-t2t-apes-2023v2.GCA_028858775.2.bigmaf.summary.bb"
2024-02-20 14:34:33.947473: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /data/tmp/81fefe459ed352c88c0799391327a7f5/0432/2938/tmp74q_1x4k/16-t2t-apes-2023v2.GCA_028858775.2.maf.gz | mafDuplicateFilter -km - | mafToBigMaf GCA_028858775.2 stdin stdout | sort -k1,1 -k2,2n'" in 1387.3652 seconds and 14.2 Mi memory
2024-02-20 14:34:33.947780: Running the command: "bedToBigBed -type=bed3+1 -as=/data/tmp/81fefe459ed352c88c0799391327a7f5/0432/2938/tmp74q_1x4k/bigMaf.as -tab /data/tmp/81fefe459ed352c88c0799391327a7f5/0432/2938/tmp74q_1x4k/bigMaf.bed /data/tmp/81fefe459ed352c88c0799391327a7f5/0432/2938/tmp74q_1x4k/GCA_028858775.2.chrom_sizes /data/tmp/81fefe459ed352c88c0799391327a7f5/0432/2938/tmp74q_1x4k/16-t2t-apes-2023v2.GCA_028858775.2.bigmaf.bb"
2024-02-20 14:34:38.966262: Successfully ran: "bedToBigBed -type=bed3+4 -as=/data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/mafSummary.as -tab /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/summary.bed /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/GCA_028858775.2.chrom_sizes /data/tmp/81fefe459ed352c88c0799391327a7f5/b0aa/0255/tmpbzkokw0r/16-t2t-apes-2023v2.GCA_028858775.2.bigmaf.summary.bb" in 8.6044 seconds and 23.8 Mi memory
2024-02-20 15:02:42.748692: Successfully ran: "bedToBigBed -type=bed3+1 -as=/data/tmp/81fefe459ed352c88c0799391327a7f5/0432/2938/tmp74q_1x4k/bigMaf.as -tab /data/tmp/81fefe459ed352c88c0799391327a7f5/0432/2938/tmp74q_1x4k/bigMaf.bed /data/tmp/81fefe459ed352c88c0799391327a7f5/0432/2938/tmp74q_1x4k/GCA_028858775.2.chrom_sizes /data/tmp/81fefe459ed352c88c0799391327a7f5/0432/2938/tmp74q_1x4k/16-t2t-apes-2023v2.GCA_028858775.2.bigmaf.bb" in 1688.8001 seconds and 247.5 Mi memory
Got message from job at time 02-20-2024 15:02:59: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-maf2bigmaf/instance-45z1s5pd/cleanup/file-c447746ee2644ad2bbff6b36dcb3e6d9/stream used 151.60% disk (55.7 GiB [59768610816B] used, 36.7 GiB [39424991053B] requested).
Finished toil run successfully.
Stopping real-time logging server.
Joining real-time logging server thread.
Exporting file:///private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/16-t2t-apes-2023v2.GCA_028858775.2.bigmaf.bb
Exporting file:///private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/16-t2t-apes-2023v2.GCA_028858775.2.bigmaf.summary.bb
Successfully deleted the job store: FileJobStore(/private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/js-bb)
cactus-maf2bigmaf has finished after 3282.590439270949 seconds