STOPPED Job While Migrating Non-CDB to PDB via Autoupgrade Mustafa, 2022-03-202022-03-20 Hi, Lately, I started to learn about autoupgrade which is the new upgrade tool for almost anything. Many thanks to Mike Dietrich for amazing blog posts ( https://mikedietrichde.com ). His blog is a real treasure. I am about to migrate some non-cdbs to pdbs and before doing that I wanted to learn my options about how I can achieve that. I will write a blog post about three ways of migrating non-cdb to pdb soon. one of them, of course, autoupgrade and I almost trying to do it for last 10 days! All my efforts has ended with a STOPPED autoupgrade job at “compatibility check” step and it drove me crazy. You must know that for 10 days I tries tens of different combinations to achieve that and every each of them failed! Finally (just a mistake) I realized what happened. Let me explain: I don’t want to go into details about autoupgrade non-cdb to pdb operation in this post, so I will just share my config file and command to run autoupgrade. Here is the test case: Oracle Database EE 19.14 on Oracle Linux 8 (virtual machine on Windows 11 host). Latest autopgrade.jar is downloaded from Oracle Support. Oracle Home is the same for both cdb (sid: cdb) and non-cdb (sid: orcl) databases. here is my autoupgrade config file: Oracle PL/SQL global.autoupg_log_dir=/home/oracle/autoupgrade upg1.log_dir=/home/oracle/autoupgrade upg1.sid=orcl upg1.source_home=/u01/app/oracle/product/19c/dbhome_1 upg1.target_cdb=cdb upg1.target_home=/u01/app/oracle/product/19c/dbhome_1 upg1.target_pdb_name=my_pdb upg1.start_time=now upg1.upgrade_node=localhost upg1.run_utlrp=yes upg1.timezone_upg=yes 123456789101112 global.autoupg_log_dir=/home/oracle/autoupgrade upg1.log_dir=/home/oracle/autoupgradeupg1.sid=orclupg1.source_home=/u01/app/oracle/product/19c/dbhome_1upg1.target_cdb=cdbupg1.target_home=/u01/app/oracle/product/19c/dbhome_1upg1.target_pdb_name=my_pdbupg1.start_time=nowupg1.upgrade_node=localhostupg1.run_utlrp=yesupg1.timezone_upg=yes then run the autoupgrade to migrade orcl db into a CDB as a pdb: Shell $ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/rdbms/admin/autoupgrade.jar -config /home/oracle/upg2.cfg -mode deploy 1 $ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/rdbms/admin/autoupgrade.jar -config /home/oracle/upg2.cfg -mode deploy I have read so many documents and blog posts about it and every time I run this command, it started Command Line Interface (CLI) of autoupgrade and when I query my job with lsj command, I got this result: Oracle PL/SQL oracle@Linux8_19c ~ $ $ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/rdbms/admin/autoupgrade.jar -config /home/oracle/upg2.cfg -mode deploy AutoUpgrade 22.1.220311 launched with default internal options Processing config file ... +--------------------------------+ | Starting AutoUpgrade execution | +--------------------------------+ 1 Non-CDB(s) will be processed Type 'help' to list console commands upg> lsj +----+-------+-----------+---------+-------+----------+-------+-------------------+ |Job#|DB_NAME| STAGE|OPERATION| STATUS|START_TIME|UPDATED| MESSAGE| +----+-------+-----------+---------+-------+----------+-------+-------------------+ | 100| orcl|NONCDBTOPDB| STOPPED|RUNNING| 22:02:12|19s ago|Compatibility check| +----+-------+-----------+---------+-------+----------+-------+-------------------+ Total jobs 1 upg> 1234567891011121314151617 oracle@Linux8_19c ~ $ $ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/rdbms/admin/autoupgrade.jar -config /home/oracle/upg2.cfg -mode deployAutoUpgrade 22.1.220311 launched with default internal optionsProcessing config file ...+--------------------------------+| Starting AutoUpgrade execution |+--------------------------------+1 Non-CDB(s) will be processedType 'help' to list console commandsupg> lsj+----+-------+-----------+---------+-------+----------+-------+-------------------+|Job#|DB_NAME| STAGE|OPERATION| STATUS|START_TIME|UPDATED| MESSAGE|+----+-------+-----------+---------+-------+----------+-------+-------------------+| 100| orcl|NONCDBTOPDB| STOPPED|RUNNING| 22:02:12|19s ago|Compatibility check|+----+-------+-----------+---------+-------+----------+-------+-------------------+Total jobs 1 upg> This is instant. I type lsj immediately when I reach to cli and I saw the upgrade job is STOPPED (status is running but operation is stopped and job is not working). So I check the log files of course and here is the error: Java 2022-03-20 22:02:14.970 WARNING Execution of the JobStageAction [oracle.upgrade.autoupgrade.dispatcher.facade.subsystems.ExecuteNonCDBToPDB] failed due to [java.lang.ArrayIndexOutOfBoundsException] - JobStage.executeStage java.lang.ArrayIndexOutOfBoundsException: 1 at oracle.upgrade.autoupgrade.noncdbtopdb.converttopdb.AbstractNonCDBToPDB.getPDBPlugInViolations(AbstractNonCDBToPDB.java:208) at oracle.upgrade.autoupgrade.noncdbtopdb.converttopdb.CheckPluginCompatibility.getReportableViolations(CheckPluginCompatibility.java:132) at oracle.upgrade.autoupgrade.noncdbtopdb.converttopdb.CheckPluginCompatibility.execCheckPlugIn(CheckPluginCompatibility.java:103) at oracle.upgrade.autoupgrade.noncdbtopdb.converttopdb.CheckPluginCompatibility.action(CheckPluginCompatibility.java:88) at oracle.upgrade.autoupgrade.noncdbtopdb.converttopdb.QuickPluginCompatibility.action(QuickPluginCompatibility.java:66) at oracle.upgrade.autoupgrade.noncdbtopdb.NonCDBToPDB.executeNonCDBToPDB(NonCDBToPDB.java:284) at oracle.upgrade.autoupgrade.dispatcher.facade.subsystems.ExecuteNonCDBToPDB.executeStage(ExecuteNonCDBToPDB.java:85) at oracle.upgrade.autoupgrade.dispatcher.v2.JobStage.executeStage(JobStage.java:71) at oracle.upgrade.autoupgrade.dispatcher.v2.RunJobDefinition.runTest(RunJobDefinition.java:129) at oracle.upgrade.autoupgrade.dispatcher.v2.def.RunAutoUpgradeJob.executeDispatcher(RunAutoUpgradeJob.java:54) at oracle.upgrade.autoupgrade.dispatcher.AutoUpgDispatcher.run(AutoUpgDispatcher.java:233) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 1234567891011121314151617181920 2022-03-20 22:02:14.970 WARNING Execution of the JobStageAction [oracle.upgrade.autoupgrade.dispatcher.facade.subsystems.ExecuteNonCDBToPDB] failed due to [java.lang.ArrayIndexOutOfBoundsException] - JobStage.executeStagejava.lang.ArrayIndexOutOfBoundsException: 1 at oracle.upgrade.autoupgrade.noncdbtopdb.converttopdb.AbstractNonCDBToPDB.getPDBPlugInViolations(AbstractNonCDBToPDB.java:208) at oracle.upgrade.autoupgrade.noncdbtopdb.converttopdb.CheckPluginCompatibility.getReportableViolations(CheckPluginCompatibility.java:132) at oracle.upgrade.autoupgrade.noncdbtopdb.converttopdb.CheckPluginCompatibility.execCheckPlugIn(CheckPluginCompatibility.java:103) at oracle.upgrade.autoupgrade.noncdbtopdb.converttopdb.CheckPluginCompatibility.action(CheckPluginCompatibility.java:88) at oracle.upgrade.autoupgrade.noncdbtopdb.converttopdb.QuickPluginCompatibility.action(QuickPluginCompatibility.java:66) at oracle.upgrade.autoupgrade.noncdbtopdb.NonCDBToPDB.executeNonCDBToPDB(NonCDBToPDB.java:284) at oracle.upgrade.autoupgrade.dispatcher.facade.subsystems.ExecuteNonCDBToPDB.executeStage(ExecuteNonCDBToPDB.java:85) at oracle.upgrade.autoupgrade.dispatcher.v2.JobStage.executeStage(JobStage.java:71) at oracle.upgrade.autoupgrade.dispatcher.v2.RunJobDefinition.runTest(RunJobDefinition.java:129) at oracle.upgrade.autoupgrade.dispatcher.v2.def.RunAutoUpgradeJob.executeDispatcher(RunAutoUpgradeJob.java:54) at oracle.upgrade.autoupgrade.dispatcher.AutoUpgDispatcher.run(AutoUpgDispatcher.java:233) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) so an ArrayIndexOutOfBoundsException is thrown. Also, in noncdb_to_pdb_orcl.log file I got this: Oracle PL/SQL 2022-03-20 22:02:14.756 INFO [688D4E] Executing SQL [/* 688D4E */select name || ':_:' || cause || ':_:' || type || ':_:' || error_number || ':_:' || line || ':_:' || message || ':_:' || status || ':_:' || action || ':_:' || con_id || ':_:' from PDB_PLUG_IN_VIOLATIONS where type = 'ERROR' and status = 'PENDING' and upper(name) = upper('MY_PDB') order by time;] in [cdb, container:null] - SQLClient.run 2022-03-20 22:02:14.968 INFO [688D4E] End - ExecuteSql.quickSQL 2022-03-20 22:02:14.969 INFO Total number of rows found in PDB_PLUG_IN_VIOLATIONS was [1] - AbstractNonCDBToPDB.getPDBPlugInViolations 12345678910111213141516 2022-03-20 22:02:14.756 INFO [688D4E] Executing SQL [/* 688D4E */select name || ':_:' || cause || ':_:' || type || ':_:' || error_number || ':_:' || line || ':_:' || message || ':_:' || status || ':_:' || action || ':_:' || con_id || ':_:'from PDB_PLUG_IN_VIOLATIONSwhere type = 'ERROR' andstatus = 'PENDING' andupper(name) = upper('MY_PDB')order by time;] in [cdb, container:null] - SQLClient.run2022-03-20 22:02:14.968 INFO [688D4E] End - ExecuteSql.quickSQL2022-03-20 22:02:14.969 INFO Total number of rows found in PDB_PLUG_IN_VIOLATIONS was [1] - AbstractNonCDBToPDB.getPDBPlugInViolations This makes me think that there might be a compatibility problem or a violation but actually there were none! I spend days to understand this 🙂 After tens of autoupgrade tests, reading hundereds of log file results, with some luck and coincidence (and my hard work of course) something got my attention. in log files I saw same familiar codes of mine! Where did I use those codes and why are they in autoupgrade logs? They are my “glogin.sql” file codes. for those who are not familiar to glogin.sql it is a initial script file for sqlplus tool in $ORACLE_HOME/sqlplus/admin directory. I use glogin.sql to set some default values like assigning db name in prompt or set SID as terminal header so I can see my current sqlplus SID all the time etc. but non of them is causing this but the guilty one is: Oracle PL/SQL set timing on 1 set timing on yes! I got “set timing on” in my glogin.sql file so I always get execution time of my sql scripts and somehow autoupgrade does not run a “set timing off” command and this cause some unexpected results. as you know, set timing on adds execution time to end of of your commands outout. probably, some scripts are assume it as a result of a query. I remove it and re-run the autoupgrade tool. everything work smooth and perfect. database is migrated as a pdb. So, I spent 10 days because of “set timing on” but at least I learned a lot and solved a problem of mine, I am proud of it 🙂 I hope this helps whoever uses glogin.sql file and set timing on in it. Wish you healthy days. Peace at the Home, Peace in the World (Mustafa Kemal ATATÃœRK) . 19c Administration Multi-tenant AbstractNonCDBToPDB.getPDBPlugInViolationsArrayIndexOutOfBoundsExceptionautoupgradeautoupgrade noncdb migrationglogin.sqlnon-cdb to pdbPDB_PLUG_IN_VIOLATIONSSTOPPED job
Hi Mustafa, can you please give us access to the logs? java -jar autoupgrade.jar -config yourconfig.cfg -zip Let us know in case you uploaded it to an SR already. Then we can fetch them from the SR. Thanks, Mike Reply
Hi Mike, Thanks for your reply. I didn’t create an SR since this was just a test case on a virtual machine (I haven’t do it on my clients system yet). I can send an email to you all logs and config if it’s ok. Reply
Hi Mustafa, I am glad you were able to figure the problem out but I am sad it took you 10 days to do it. We have fixed the problem and it will be in the next kit due out soon! – Joe – Reply
Hi Joseph, thanks for your reply. I am always happy when I figure out things by myself so 10 days is not a problem 🙂 thank you very much for your interest and glad to hear it will be fixed. Reply